Systems and methods for target tracking

ABSTRACT

The present invention provides methods and systems of tracking a target with an imaging device onboard an unmanned aerial vehicle. The method may comprise identifying one or more targets in a first image captured by the imaging device, receiving a user selection of a target to be tracked from the one or more targets in the one or more images, generating target information for the target to be tracked based on the user selection of the target, detecting a deviation of a detected position and/or a detected size of the target in a second image captured by the imaging device from the selected position and/or the selected size of the target in the first image and adjusting at least one of the UAV and the carrier, to reduce the determined deviation to maintain the target substantially within the field of view of the image device, based on the deviation.

CROSS-REFERENCE

This application is a continuation application of U.S. application Ser.No. 15/289,842, filed on Oct. 10, 2016, which is a continuationapplication of U.S. application Ser. No. 15/182,553, filed on Jun. 14,2016, which is a continuation application of U.S. application Ser. No.14/845,894, filed on Sep. 4, 2015, which is a continuation applicationof U.S. application Ser. No. 14/471,954, filed on Aug. 28, 2014, nowU.S. Pat. No. 9,164,506, issued Oct. 20, 2015, which is a continuationapplication of International Application No. PCT/CN2014/083315, filed onJul. 30, 2014, the contents of which are hereby incorporated byreference in their entirety.

BACKGROUND OF THE INVENTION

Aerial vehicles such as unmanned aerial vehicles (UAVs) can be used forperforming surveillance, reconnaissance, and exploration tasks formilitary and civilian applications. Such aerial vehicles may carry apayload configured to perform a specific function such as capturingimages of surrounding environment.

In some instances, it may be desirable for aerial vehicles to track aspecific target. For small-sized aerial vehicles, such tracking istraditionally achieved via control commands from a user-operated remotecontrol terminal or device. Such manual tracking control may becomedifficult in certain circumstances, such as when the movable object ortarget is moving quickly or when the movable object is at leastpartially blocked from view of the user. Furthermore, the attentionnecessary for such manual tracking typically requires a dedicated userthat controls a camera that onboard the aerial vehicle separate from auser that controls the navigation of the aerial vehicle, therebyincreasing the cost for aerial photography and other applications of theaerial vehicles.

SUMMARY OF THE INVENTION

In some instances, it may be desirable for aerial vehicles to track aspecific target. Thus, a need exists for improved UAV tracking methodsand systems that provides automatic or semi-automatic tracking of atarget, thereby relieving operators of the aerial vehicles of manuallytracking the targets. The present invention provides systems, methods,and devices related to target tracking by UAVs. The UAV may beconfigured to receive target information from a control terminal relatedto a target to be tracked by an imaging device coupled to the UAV. Thetarget information may be used by the UAV to automatically track thetarget so as to maintain predetermined position and/or size of thetarget within one or more images captured by the imaging device. Anydescription of tracking may include visual tracking by the imagingdevice. The control terminal may be configured to display images fromthe imaging device as well as allowing user input related to the targetinformation.

According to an aspect of the present invention, a method forcontrolling an unmanned aerial vehicle (UAV) is provided. The methodcomprises: receiving, from a remote user, one or more navigationcommands to move the UAV along a flight path; receiving, from the remoteuser, target information of a target to be tracked by an imaging deviceon the UAV; and tracking the target according to the target informationby automatically adjusting at least one of the UAV or the imaging devicewhile the UAV moves along the flight path according to the one or morenavigation commands from the remote user.

According to another aspect of the present invention, an unmanned aerialvehicle (UAV) with tracking capabilities is provided. The UAV comprises:one or more receivers, individually or collectively, configured toreceive from a remote user (1) one or more navigation commands to movethe UAV along a flight path, and (2) target information of a target tobe tracked by an imaging device on the UAV; and one or more processors,individually or collectively, configured to track the target accordingto the target information by automatically adjusting at least one of theUAV or the imaging device while the UAV moves along the flight pathaccording to the one or more navigation commands from the remote user.

According to another aspect of the present invention, s system forcontrolling an unmanned aerial vehicle (UAV) is provided. The systemcomprises: one or more receivers, individually or collectively,configured to receive from a remote user (1) one or more navigationcommands to move the UAV along a flight path, and (2) target informationof a target to be tracked by an imaging device on the UAV; and one ormore processors, individually or collectively, configured to track thetarget according to the target information by automatically adjusting atleast one of the UAV or the imaging device while the UAV moves along theflight path according to the one or more navigation commands from theremote user.

In some embodiments, the imaging device includes a camera or acamcorder.

In some embodiments, the one or more navigation commands are adapted tocontrol a speed, position or attitude or the UAV.

In some embodiments, the target is substantially stationary relative toa reference object.

In some embodiments, the target is moving relative to a referenceobject.

In some embodiments, the target information includes initial targetinformation.

In some embodiments, the initial target information includes an initialposition or an initial size of the target within an image captured bythe imaging device.

In some embodiments, the target information includes target typeinformation.

In some embodiments, tracking the target according to the targetinformation further includes identifying, based on the target typeinformation, the target to track from within one or more images capturedby the imaging device using an image recognition algorithm.

In some embodiments, the target type information includes color,texture, or pattern information.

In some embodiments, the target information includes expected targetinformation.

In some embodiments, the expected target information includes anexpected position or an expected size of the target within an imagecaptured by the imaging device.

In some embodiments, the expected size of the target is the same as aninitial size of the target.

In some embodiments, the expected position of the target is the same asan initial position of the target.

In some embodiments, tracking the target according to the targetinformation includes maintaining, within a predetermined degree oftolerance, the expected position, or the expected size of the targetwithin one or more images captured by the imaging device.

In some embodiments, the imaging device is coupled to the UAV via acarrier configured to permit the imaging device to move relative to theUAV.

In some embodiments, the carrier is configured to permit the imagingdevice to rotate around at least two axes relative to the UAV.

In some embodiments, tracking the target according to the targetinformation includes automatically adjusting at least one of the UAV,the carrier, or the imaging device while the UAV moves along the flightpath according to the one or more navigation commands from the remoteuser.

In some embodiments, the target information includes expected targetinformation and tracking the target according to the target informationcomprises: determining current target information of the target based onone or more images captured by the imaging device; detecting a deviationof the current target information from the expected target information;and calculating an adjustment to the UAV, the carrier, or the imagingdevice so as to substantially correct the deviation.

In some embodiments, the deviation is related to a change in position ofthe target and the calculated adjustment is related to an angularvelocity for the UAV.

In some embodiments, the angular velocity is relative to a yaw axis ofthe UAV.

In some embodiments, the angular velocity is relative to a pitch axis ofthe UAV.

In some embodiments, the deviation is related to a change in position ofthe target and the calculated adjustment is related to an angularvelocity for the imaging device relative to the UAV.

In some embodiments, the calculated adjustment is used to generatecontrol signals for the carrier so as to cause the imaging device tomove relative to the UAV.

In some embodiments, the deviation is related to a change in size of thetarget and the adjustment is related to a linear velocity for the UAV.

In some embodiments, the deviation is related to a change in size of thetarget and the adjustment is related to one or more parameters of theimaging device.

In some embodiments, the one or more parameters of the imaging deviceinclude focal length, zoom, or focus.

In some embodiments, the calculated adjustment is limited to apredetermined range.

In some embodiments, the predetermined range corresponds to apredetermined range of control lever amount of a control system.

In some embodiments, the control system includes a flight control systemfor the UAV or a control system for the carrier.

In some embodiments, a warning signal is provided if the calculatedadjustment falls outside the predetermined range.

In some embodiments, tracking the target comprises comparing thecalculated adjustment to a predetermined maximum threshold value andproviding the predetermined maximum threshold value if the calculatedadjustment exceeds the predetermined maximum threshold value.

In some embodiments, the predetermined maximum threshold value includesa maximum angular velocity or a maximum linear velocity for the UAV orthe imaging device.

In some embodiments, tracking the target comprises comparing thecalculated adjustment to a predetermined minimum threshold value andproviding the predetermined minimum threshold value if the calculatedadjustment is less than the predetermined minimum threshold value.

In some embodiments, the predetermined minimum threshold value includesa minimum angular velocity or a minimum linear velocity for the UAV orthe imaging device.

In some embodiments, the target information is received from a remotecontrol device accessible to the remote user.

In some embodiments, the one or more navigation commands are receivedfrom the same remote control device.

In some embodiments, the one or more navigation commands are receivedfrom a different remote control device.

In some embodiments, the remote control device is configured to receiveuser input from a touchscreen, joystick, keyboard, mouse, or stylus.

In some embodiments, the remote control device is configured to receiveuser input from a wearable device.

In some embodiments, the remote control device is configured to: receiveone or more images captured by the imaging device from the UAV; displaythe one or more images; receive a user selection of a target from withina displayed image; generate the target information of the target basedon the user selection of the target; and transmit the target informationto the UAV.

In some embodiments, the remote control device is further configured togenerate the one or more navigation commands based on user input and totransmit the one or more navigation commands to the UAV.

In some embodiments, the remote control device is further configured toreceive tracking information related to the target and to display theone or more images with the tracking information.

According to an aspect of the present invention, an unmanned aerialvehicle (UAV) with tracking capabilities is provided. The UAV comprises:one or more receivers, individually or collectively, configured toreceive, from a remote user, user-specified target information of atarget to be tracked by an imaging device on the UAV, the user-specifiedtarget information including a predetermined position or a predeterminedsize of the target within an image captured by the imaging device, theimaging device coupled to the UAV via a carrier configured to permit theimaging device to move relative to the UAV; and one or more processors,individually or collectively, configured to: detect a deviation from thepredetermined position or the predetermined size of the target based onone or more images captured by the imaging device; and generate commandsto automatically adjust the UAV, the carrier, or the imaging device soas to substantially correct the detected deviation from thepredetermined position or the predetermined size of the target.

According to another aspect of the present invention, a system forcontrolling an unmanned aerial vehicle (UAV) is provided. The systemcomprises: one or more receivers, individually or collectively,configured to receive, from a remote user, user-specified targetinformation of a target to be tracked by an imaging device on the UAV,the user-specified target information including a predetermined positionor a predetermined size of the target within an image captured by theimaging device, the imaging device coupled to the UAV via a carrierconfigured to permit the imaging device to move relative to the UAV; andone or more processors, individually or collectively, configured to:detect a deviation from the predetermined position or the predeterminedsize of the target based on one or more images captured by the imagingdevice; and generate commands to automatically adjust the UAV, thecarrier, or the imaging device so as to substantially correct thedetected deviation from the predetermined position or the predeterminedsize of the target.

According to another aspect of the present invention, a method forcontrolling an unmanned aerial vehicle (UAV) is provided. The methodcomprises: receiving, from a remote user, user-specified targetinformation of a target to be tracked by an imaging device on the UAV,the user-specified target information including a predetermined positionor predetermined size of the target within an image captured by theimaging device, the imaging device coupled to the UAV via a carrierconfigured to permit the imaging device to move relative to the UAV;detecting, by a processor onboard the UAV, a deviation from thepredetermined position or the predetermined size of the target based onone or more images captured by the imaging device; and automaticallyadjusting the UAV, the carrier, or the imaging device so as tosubstantially correct the detected deviation from the predeterminedposition or the predetermined size of the target.

In some embodiments, the imaging device includes a camera or acamcorder.

In some embodiments, the method further comprises receiving, from theremote user, one or more commands adapted to control a speed, position,orientation or attitude or the UAV.

In some embodiments, the method further comprises receiving, from theremote user, one or more commands adapted to control a speed, position,orientation or attitude or the carrier.

In some embodiments, the method further comprises receiving, from theremote user, one or more commands adapted to control one or moreoperational parameters of the imaging device.

In some embodiments, the one or more operational parameters of theimaging device include focal length, zoom level, imaging mode, imageresolution, focus, depth of field, exposure, lens speed, or field ofview.

In some embodiments, the carrier is configured to permit the imagingdevice to rotate around at least one axes relative to the UAV.

In some embodiments, the carrier is configured to permit the imagingdevice to rotate around at least two axes relative to the UAV.

In some embodiments, the target information of the target furtherincludes target type information.

In some embodiments, the target type information includes a color ortexture of the target.

In some embodiments, the predetermined position of the target includesan initial position or an expected position of the target.

In some embodiments, the predetermined size of the target includes aninitial size or an expected size of the target.

In some embodiments, detecting the deviation from the predeterminedposition or the predetermined size of the target comprises comparing aposition or size of the target within the one or more images captured bythe imaging device with the predetermined position or predeterminedsize, respectively.

In some embodiments, adjusting the UAV, the carrier, or the imagingdevice comprises calculating an adjustment to the UAV, the carrier, orthe imaging device so as to substantially correct the deviation.

In some embodiments, the deviation is related to a change in position ofthe target and the adjustment is related to an angular velocity for theUAV.

In some embodiments, the angular velocity is relative to a yaw axis ofthe UAV.

In some embodiments, the angular velocity is relative to a pitch axis ofthe UAV.

In some embodiments, the deviation is related to a change in position ofthe target and the adjustment is related to an angular velocity for theimaging device relative to the UAV.

In some embodiments, the adjustment is used to generate control signalsfor the carrier so as to cause the imaging device to move relative tothe UAV.

In some embodiments, the angular velocity is relative to a yaw axis ofthe imaging device.

In some embodiments, the angular velocity is relative to a pitch axis ofthe imaging device.

In some embodiments, the deviation is related to a change in size of thetarget and the adjustment is related to a linear velocity for the UAV.

In some embodiments, the deviation is related to a change in size of thetarget and the adjustment is related to one or more parameters of theimaging device.

In some embodiments, the one or more parameters of the imaging deviceinclude focal length, zoom level, imaging mode, image resolution, focus,depth of field, exposure, lens speed, or field of view.

In some embodiments, the calculated adjustment is limited to apredetermined range.

In some embodiments, the predetermined range corresponds to apredetermined range of control lever amount of a control system.

In some embodiments, the control system includes a navigation controlsystem for the UAV or a control system for the carrier.

In some embodiments, the method further comprises providing a warningsignal if the adjustment falls outside the predetermined range.

In some embodiments, the warning signal is used to provide an audio orvisual signal.

In some embodiments, the warning signal is used to provide a kineticsignal.

In some embodiments, the method further comprises transmitting, insubstantially real-time, images captured by the imaging device to aremote user device accessible to the remote user.

In some embodiments, the remote user device comprises a display fordisplaying the images captured by the imaging device.

In some embodiments, the remote user device comprises an input devicefor providing the target information.

In some embodiments, the input device includes a touchscreen, joystick,keyboard, mouse, or stylus.

In some embodiments, the input device includes a wearable device.

In some embodiments, the target information is provided based on thetransmitted images.

In some embodiments, the method further comprises providing, insubstantially real-time, tracking information of the target to theremote user device.

In some embodiments, the remote user device is configured to: receive auser selection of the target from within one or more images displayed onthe remote user device; and generate the target information of thetarget based on the user selection of the target.

According to another aspect of the present invention, a method forcontrolling an unmanned aerial vehicle (UAV) is provided. The methodcomprises: displaying, via a display, one or more images captured by animaging device coupled to the UAV in substantially real-time; receiving,via an input device, a user selection of a target from within at leastone of the one or more images being displayed in substantiallyreal-time; generating target information of the target based at least inpart on the user selection of the target; and providing the targetinformation to the UAV so as to allow the UAV autonomously track thetarget according to the target information.

According to another aspect of the present invention, a system forcontrolling an unmanned aerial vehicle (UAV) is provided. The systemcomprises: a display configured to display one or more images capturedby an imaging device coupled to the UAV; an input device configured toreceive a user selection of a target from within at least one of the oneor more images being displayed on the display; one or more processors,individually or collectively, configured to generate target informationof the target based at least in part on the user selection of thetarget; and a transmitter configured to provide the target informationto the UAV so as to allow the UAV autonomously track the targetaccording to the target information.

According to another aspect of the present invention, an apparatus forcontrolling an unmanned aerial vehicle (UAV) is provided. The apparatuscomprises: a display configured to display one or more images capturedby an imaging device coupled to the UAV; an input device configured toreceive a user selection of a target from within at least one of the oneor more images being displayed on the display; one or more processors,individually or collectively, configured to generate target informationof the target based at least in part on the user selection of thetarget; and a transmitter configured to provide the target informationto the UAV so as to allow the UAV autonomously track the targetaccording to the target information.

In some embodiments, the target information includes initial targetinformation.

In some embodiments, the initial target information includes an initialposition or an initial size of the target within an image captured bythe imaging device.

In some embodiments, the initial target information is generated basedon the user selection of the target.

In some embodiments, the target information includes target typeinformation.

In some embodiments, the target type information includes color,texture, or pattern information.

In some embodiments, the target type information is generated based onthe user selection of the target.

In some embodiments, the target information includes expected targetinformation.

In some embodiments, the expected target information is generated basedon the user selection of the target.

In some embodiments, the expected target information includes anexpected position or an expected size of the target within an imagecaptured by the imaging device.

In some embodiments, the target information does not include expectedtarget information.

In some embodiments, the input device includes a touchscreen, joystick,keyboard, mouse, stylus, or wearable device.

In some embodiments, the user selection of the target is achieved by auser selecting an area of the at least one of the one or more imagesbeing displayed on the display, the selected area corresponding to thetarget.

In some embodiments, the user selection of the target is achieved by auser directly touching an area of the at least one of the one or moreimages being displayed on the display, the touched area corresponding tothe target.

In some embodiments, the user selects the area using a stylus, mouse,keyboard, or a wearable device.

In some embodiments, selecting the area includes touching, swiping,circling, or clicking in the area.

In some embodiments, the one or more processors, individually orcollectively, are further configured to display, on the display, theselected target with a selection indicator in response to the userselection of the target, the selection indicator indicating that thetarget has been selected by the user.

In some embodiments, the one or more processors, individually orcollectively, are further configured to receive tracking informationrelated to the target and, based on the tracking information, displaythe selected target with a tracking indicator within one or moresubsequent images captured by the imaging device, the tracking indicatorindicating, in substantially real-time, that the target is being trackedby the UAV according to the target information.

According to another aspect of the present invention, a method forcontrolling an unmanned aerial vehicle (UAV) is provided. The methodcomprises: receiving, in substantially real-time, one or more imagescaptured by an imaging device coupled to the UAV; receivinguser-specified target information of a target including a predeterminedposition or a predetermined size of a target within an image from theone or more images; providing the user-specified target information tothe UAV; and displaying the one or more images and a tracking indicatorassociated with the target within the one or more images, the trackingindicator indicating that the target is being tracked by the UAV insubstantially real-time according to the user-specified targetinformation.

According to another aspect of the present invention, an apparatus forcontrolling an unmanned aerial vehicle (UAV) is provided. The apparatuscomprises: a receiver configured to receive, in substantially real-time,one or more images captured by an imaging device coupled to the UAV; aninput device configured to receive user-specified target information ofa target including a predetermined position or a predetermined size of atarget within an image from the one or more images; a transmitter forprovide the user-specified target information to the UAV; and a displayconfigured to display the one or more images and a tracking indicatorassociated with the target within the one or more images, the trackingindicator indicating that the target is being tracked by the UAV insubstantially real-time according to the user-specified targetinformation.

According to another aspect of the present invention, a system forcontrolling an unmanned aerial vehicle (UAV) is provided. The systemcomprises: a receiver configured to receive, in substantially real-time,one or more images captured by an imaging device coupled to the UAV; aninput device configured to receive user-specified target information ofa target including a predetermined position or a predetermined size of atarget within an image from the one or more images; a transmitter forprovide the user-specified target information to the UAV; and a displayconfigured to display the one or more images and a tracking indicatorassociated with the target within the one or more images, the trackingindicator indicating that the target is being tracked by the UAV insubstantially real-time according to the user-specified targetinformation.

In some embodiments, the input device is further configured to receiveone or more commands adapted to control a speed, position, orientationor attitude or the UAV, or one or more operational parameters of theimaging device.

In some embodiments, the imaging device is coupled to the UAV via acarrier configured to permit the imaging device to rotate relative tothe UAV along at least one axis and wherein the input device is furtherconfigured to receive one or more commands adapted to control a speed,position, orientation, or attitude of the carrier.

In some embodiments, the carrier is configured to permit the imagingdevice to rotate around at least two axes relative to the UAV.

In some embodiments, the target is tracked by the imaging deviceaccording to the target information via automatic adjustment to at leastone of the UAV, the carrier, or the imaging device.

In some embodiments, a second input device is included and configured toreceive one or more commands adapted to control a speed, position,orientation or attitude or the UAV, or one or more operationalparameters of the imaging device.

In some embodiments, the imaging device is coupled to the UAV via acarrier configured to permit the imaging device to rotate relative tothe UAV along at least one axis and wherein the system further comprisea second input device configured to receive one or more commands adaptedto control a speed, position, orientation or attitude of the carrier.

In some embodiments, the one or more operational parameters of theimaging device include focal length, zoom level, imaging mode, imageresolution, focus, depth of field, exposure, lens speed, or field ofview.

In some embodiments, the target is tracked by the imaging deviceaccording to the target information via automatic adjustment to at leastone of the UAV, the carrier, or the imaging device.

In some embodiments, the predetermined position includes an initialposition of the target.

In some embodiments, the predetermined size includes an initial size ofthe target.

In some embodiments, the predetermined position includes an expectedposition of the target.

In some embodiments, the predetermined size includes an expected size ofthe target.

In some embodiments, the target information further includes target typeinformation.

In some embodiments, the target information is generated based on a userselection of the target.

In some embodiments, the tracking indicator includes a geometric shape,a check mark, or an arrow.

In some embodiments, the geometric shape includes a circle, a rectangle,or a triangle.

In some embodiments, the target is tracked by the imaging deviceaccording to the target information via automatic adjustment to at leastone of the UAV or the imaging device.

It shall be understood that different aspects of the invention can beappreciated individually, collectively, or in combination with eachother. Various aspects of the invention described herein may be appliedto any of the particular applications set forth below or for any othertypes of movable objects. Any description herein of aerial vehicles,such as unmanned aerial vehicles, may apply to and be used for anymovable object, such as any vehicle. Additionally, the systems, devices,and methods disclosed herein in the context of aerial motion (e.g.,flight) may also be applied in the context of other types of motion,such as movement on the ground or on water, underwater motion, or motionin space.

Additional aspects and advantages of the present disclosure will becomereadily apparent to those skilled in this art from the followingdetailed description, wherein only exemplary embodiments of the presentdisclosure are shown and described, simply by way of illustration of thebest mode contemplated for carrying out the present disclosure. As willbe realized, the present disclosure is capable of other and differentembodiments, and its several details are capable of modifications invarious obvious respects, all without departing from the disclosure.Accordingly, the drawings and description are to be regarded asillustrative in nature, and not as restrictive.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1 illustrates an exemplary target tracking system, in accordancewith embodiments.

FIG. 2 illustrates exemplary flow of image-related data among componentsin a tracking system, in accordance with embodiments.

FIG. 3 illustrates exemplary flow of control data among components in atracking system, in accordance with embodiments.

FIG. 4 illustrates an exemplary process for implementing targettracking, in accordance with embodiments.

FIG. 5 illustrates an exemplary configuration of a movable object,carrier, and payload, in accordance with embodiments.

FIG. 6 illustrates an exemplary tracking method for maintaining anexpected position of a target, in accordance with embodiments.

FIG. 7 illustrates an exemplary tracking method for maintaining anexpected size of a target, in accordance with embodiments.

FIG. 8 illustrates another exemplary process for implementing targettracking, in accordance with embodiments.

FIG. 9 illustrates an exemplary process for controlling a movable objectto navigate and track, in accordance with embodiments.

FIG. 10 illustrates an exemplary process for selecting a target, inaccordance with embodiments.

FIG. 11 illustrates an exemplary process for viewing a tracked target,in accordance with embodiments.

FIG. 12 illustrates an exemplary control terminal for controlling amovable object, in accordance with embodiments.

FIGS. 13A-C illustrate exemplary methods for selecting a target using auser interface, in accordance with some embodiments.

FIG. 14 illustrates a UAV, in accordance with embodiments.

FIG. 15 illustrates a movable object including a carrier and a payload,in accordance with embodiments.

FIG. 16 illustrates an exemplary system for tracking a movable object,in accordance with embodiments.

FIG. 17 illustrates an exemplary system for controlling a movableobject, in accordance with embodiments.

FIG. 18 illustrates an exemplary use case for the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides systems, methods, and devices related totarget tracking by unmanned aerial vehicles (UAVs). A UAV may beconfigured to receive target information from a control terminal relatedto a target to be tracked by an imaging device coupled to the UAV. Thetarget information may be used by the UAV to automatically cause theimaging device to track the target so as to maintain predeterminedposition and/or size of the target within one or more images captured bythe imaging device. The tracking of the target may be performed whilethe UAV is controlled to navigate according user commands and/orpredetermined navigation paths. The control terminal may be configuredto display images from the imaging device as well as allowing user inputrelated to the target information.

Using the tracking methods and systems provided herein, a single usercan control both the navigation of a UAV and tracking of a targetsubstantially concurrently without the help of an extra person. The usercan utilize a user interface of a control terminal to specify the targetto track and/or the type of target to track. Such user-specified targetinformation may be transmitted to the UAV, which can autonomously trackthe target, for example, using an imaging device onboard the UAV. Imagescaptured by the imaging device (e.g., pictures and/or videos) can betransmitted in real time to the control terminal for display, playback,storage, or other purposes. The user may also change or adjust thetarget to track in real time using the control terminal. Advantageously,such autonomous tracking can be performed by the UAV while the userengages in other activities such as controlling the navigation of theUAV, or other activities.

For instance, a user can configure the UAV to track herself as sheengages in a variety of activities such as hiking or biking. She mayspecify herself as the tracking target for the UAV using the userinterface provided by the control terminal. For example, she may selectherself as the target from an image displayed on a user interface of thecontrol terminal, for example, using a touchscreen. Once the targetinformation is transmitted to the UAV, the user can be relieved from thelow-level operations associated with manual tracking of a target such asadjusting the UAV, carrier or imaging device. Instead, she can focus onother activities such as biking while the UAV automatically tracks inreal time her based on the provided target information using the methodsprovided herein. For instance, the attitude, position, velocity, zoom,and other aspects of the UAV and/or the imaging device can beautomatically adjusted to ensure that the user maintains a designatedposition and/or size within the images captured by the imaging device.Images captured during the tracking process (e.g., videos or pictures)may be streamed to the control terminal in real time or substantiallyreal time for display, playback, storage, or other purposes. All of theabove can be achieved by one person in a relatively painless manner,making it easier for users achieve previously difficult-to-achievetasks.

Such tracking methods and systems as described herein advantageouslyfacilitate the automation of the low-level control portion of thetracking process so as to reduce the efforts required and the errorsresulting from manual tracking. At the same time, the tracking methodsand system described herein still allows users to maintain, if desired,high-level control of the tracking process (e.g., by specifying the typeof target to track).

FIG. 1 illustrates an exemplary target tracking system 100, inaccordance with embodiments. The system 100 includes a movable object101 and a control terminal 112. The system 100 may be used to track oneor more targets 116. Although the movable object 101 is depicted as anunmanned aerial vehicle (UAV), this depiction is not intended to belimiting, and any suitable type of movable object can be used, asdescribed herein. One of skill in the art would appreciate that any ofthe embodiments described herein in the context of aircraft systems canbe applied to any suitable movable object.

In some embodiments, the movable object 101 can include a carrier 102and a payload 104. The carrier 102 may permit the payload 104 to moverelative to the movable object 101. For instance, the carrier 102 maypermit the payload 104 to rotate around one, two, three, or more axes.Alternatively or additionally, the carrier 102 may permit the payload104 to move linearly along one, two, three, or more axes. The axes forthe rotational or translational movement may or may not be orthogonal toeach other.

In some embodiments, the payload 104 may be rigidly coupled to orconnected with the movable object 101 such that the payload 104 remainssubstantially stationary relative to the movable object 101. Forexample, the carrier 102 that connects the movable object 101 and thepayload 104 may not permit the payload 104 to move relative to themovable object 101. Alternatively, the payload 104 may be coupleddirectly to the movable object 101 without requiring a carrier.

In some embodiments, the payload 104 can include one or more sensors forsurveying or tracking one or more targets 116. Examples of such apayload may include an image capturing device or imaging device (e.g.,camera or camcorder, infrared imaging device, ultraviolet imagingdevice, or the like), an audio capture device (e.g., a parabolicmicrophone), an infrared imaging device, or the like. Any suitablesensor(s) can be incorporated into the payload 104 to capture anyvisual, audio, electromagnetic, or any other desirable signals. Thesensors can provide static sensing data (e.g., a photograph) or dynamicsensing data (e.g., a video). The sensors may capture sensing datacontinuously in real time or at high frequencies.

In various embodiments, the target 116 being tracked the movable object101 can include any natural or man-made objects or structures suchgeographical landscapes (e.g., mountains, vegetation, valleys, lakes, orrivers), buildings, vehicles (e.g., aircrafts, ships, cars, trucks,buses, vans, or motorcycle). The target 116 can also include livesubjects such as people or animals. The target 116 may be moving orstationary relative to any suitable reference frame. The reference framecan be a relatively fixed reference frame (e.g., the surroundingenvironment, or earth). Alternatively, the reference frame can be amoving reference frame (e.g., a moving vehicle). In various embodiments,the target 116 may include a passive target or an active target. Anactive target may be configured to transmit information about thetarget, such as the target's GPS location, to the movable object.Information may be transmitted to the movable object via wirelesscommunication from a communication unit of the active target to acommunication unit of the movable object. Examples of an active targetcan include a friendly vehicle, building, troop, or the like. A passivetarget is not configured to transmit information about the target.Examples of a passive target can include a neutral or hostile vehicle,building, troop, and the like.

The movable object 101 can be configured to receive, and the controlterminal 112 can be configured to provide control data. The control datacan be used to control, directly or indirectly, aspects of the movableobject 101. In some embodiments, the control data can include navigationcommands for controlling navigational parameters of the movable objectsuch as the position, speed, orientation, or attitude of the movableobject 101. The control data can be used to control flight of a UAV. Thecontrol data may affect operation of one or more propulsion units 106that may affect the flight of the UAV. In other cases, the control datacan include commands for controlling individual components of themovable object 101. For instance, the control data may includeinformation for controlling the operations of the carrier 102. Forexample, the control data may be used to control an actuation mechanismof the carrier 102 so as to cause angular and/or linear movement of thepayload 104 relative to the movable object 101. As another example, thecontrol data may be used to control the movement of the carrier 102without the payload. As another example, the control data may be used toadjust one or more operational parameters for the payload 104 such astaking still or moving pictures, zooming in or out, turning on or off,switching imaging modes, change image resolution, changing focus,changing depth of field, changing exposure time, changing speed of lens,changing viewing angle or field of view, or the like. In otherembodiments, the control data may be used to control a sensing system(not show), communication system (not shown), and the like, of themovable object 101.

In some embodiments, the control data from the control terminal 112 caninclude target information. In some cases, the target information caninclude characteristics of a specific target such as an initial position(e.g., coordinates) and/or size of a target within one or more imagescaptured by an imaging device carried by the movable object 101.Additionally or alternatively, the target information can include targettype information such as characteristics of a type or category oftargets including color, texture, pattern, size, shape, dimension, andthe like. Target information can include data representation of an imageof the target. This may include an image of the target in a field ofview. Field of view may be defined or encompassed by the images capturedby the imaging device.

Target information can also include expected target information. Theexpected target information specifies the characteristics that thetarget being tracked is expected to meet in the images captured by theimaging device. The expected target information may be used to adjustthe movable object, carrier and/or imaging device so that the targetbeing tracked maintains an appearance in one or more images according tothe expected target information. For example, the target may be trackedso as to maintain an expected position and/or size within one or moreimages captured by the imaging device. For example, the expectedposition of the tracked target may be near the center of the image oroff-center. The expected size of the tracked target may be around acertain number of pixels. The expected target information may or may notbe the same as the initial target information. In various embodiments,expected target information may or may not be provided by the controlterminal. For example, expected target information may be hardcoded inthe control logic executed by a processing unit onboard the movableobject, stored in a data store local and/or remote to the movableobject, or obtained from other suitable sources.

In some embodiments, the target information (including specific targetinformation and target type information) may be generated at least inpart on user input at the control terminal 112. Additionally oralternatively, the target information may be generated based on datafrom other sources. For example, target type information may be derivedbased on previous images and/or data extracted from local or remote datastores. The images could have been previously captured by the imagingdevice 104 coupled to the movable object 101 or other devices. Theimages could be computer-generated. Such target type information may beselected by the user and/or provided automatically by default to themovable object.

The target information may be used by the movable object 101 to trackone or more targets 116. The tracking and any other related dataprocessing may be performed at least in part by one or more processorsonboard the movable object 101. In some embodiments, the targetinformation can be used to identify, by the movable object, the target116 to be tracked. Such identification of the target may be performedbased on the initial target information including the specificcharacteristics of a particular target (e.g., initial coordinates of thetarget within an image captured by the movable object), or generalcharacteristics of a type of target (e.g., color and/or texture of thetarget(s) to be tracked). In some cases, target identification caninvolve any suitable image recognition and/or matching algorithms. Insome embodiments, target identification includes comparing two or moreimages to determine, extract, and/or match features contained therein.

Once a target is identified, expected target information can be used todetect a deviation from expected characteristics of the target such asexpected position and/or size. In some embodiments, current targetcharacteristics or information can be determined based on one or moreimages captured by the movable object. The current target informationcan be compared with the expected target information provided by thecontrol terminal to determine the deviation therefrom. A change inposition of the target may be detected by comparing coordinates of thetarget (e.g., the coordinates of a center point of the target) within animage to the coordinates of the expected target position. A change insize of the target may be detected by comparing the size of the area(e.g., in pixels) covered by the target with the expected target size.In some embodiments, a change in size may be detected by detecting anorientation, boundaries, or other characteristics of the target.

Based at least in part on the detected deviation, control signals may begenerated (e.g., by one or more processors onboard the movable object)that cause adjustment that substantially corrects the detecteddeviation. As such, the adjustment may be used to substantially maintainone or more expected target characteristics (e.g., target positionand/or size) within the images captured by the movable object. In someembodiments, the adjustment may be performed in substantially real timeas the movable object is executing user-provided navigation commands(e.g., hovering or moving) and/or predetermined navigation paths. Theadjustment may also be performed in substantially real time as theimaging device is capturing one or more images. In some embodiments, theadjustment may be generated based on other information such as sensingdata acquired by one or more sensors onboard the movable object (e.g.,proximity sensor, or GPS sensor). For example, position information ofthe target being tracked may be obtained by a proximity sensor and/orprovided by the target itself (e.g., GPS location). Such positioninformation may be used, in addition to the detected deviation, togenerate the adjustment.

The adjustment may pertain to the movable object, the carrier, and/orthe payload (e.g., imaging device). For example, the adjustment maycause the movable object and/or the payload (e.g., imaging device) tochange its position, attitude, orientation, angular and/or linearvelocity, angular and/or linear velocity, and the like. The adjustmentmay cause the carrier to move the payload (e.g., imaging device)relative to the movable object such as around or along one, two, three,or more axes. Furthermore, the adjustment may include adjustment to thezoom, focus, or other operational parameters of the payload (e.g.,imaging device) itself (e.g., zoom in/out).

In some embodiments, the adjustment may be generated based at least inpart on the type of detected deviation. For example, a deviation fromthe expected target position may require rotation of the movable objectand/or the payload (e.g., via the carrier) around one, two, or threerotational axes. As another example, a deviation from the expectedtarget size may require translational movement of the movable objectalong a suitable axis and/or changes to the zoom of the imaging device(e.g., zoom in or out). For example, if the current or actual targetsize is smaller than the expected target size, the movable object mayneed to be moved closer to the target and/or the imaging device may needto be zoomed in to the target. On the other hand, if the current oractual target size is larger than the expected target size, the movableobject may need to be moved farther away from the target and/or theimaging device may need to be zoomed out from the target.

In various embodiments, the adjustment to substantially correct thedeviation from expected target information may be achieved bycontrolling one or more controllable objects such as the movable object,the carrier, the imaging device, or any combination thereof via controlsignals. In some embodiments, the controllable objects may be selectedto implement an adjustment and the corresponding control signals may begenerated based at least in part on the configurations or settings ofthe controllable objects. For example, an adjustment that involvesrotation around two axes (e.g., yaw and pitch) may be achieved solely bycorresponding rotation of the movable object around the two axes if theimaging device is rigidly coupled to the movable object and hence notpermitted to move relative to the movable object. Such may be the casewhen the imaging device is directly coupled to the movable object, orwhen the imaging device is coupled to the movable object via a carrierthat does not permit relative movement between the imaging device andthe movable object. The same two-axis adjustment may be achieved bycombining adjustment to both the movable object and the carrier if thecarrier permits the imaging device to rotate around at least one axisrelative to the movable object. In this case, the carrier can becontrolled to implement the rotation around one or two of the two axesrequired for the adjustment and the movable object can be controlled toimplement the rotation around one or two of the two axes. For example,the carrier may include a one-axis gimbal that allows the imaging deviceto rotate around one of the two axes required for adjustment while therotation around the remaining axis is achieved by the movable object.Alternatively, the same two-axis adjustment may be achieved by thecarrier alone if the carrier permits the imaging device to rotate aroundtwo or more axes relative to the movable object. For instance, thecarrier may include a two-axis or three-axis gimbal.

As another example, an adjustment to correct a change in size of thetarget may be achieved by controlling the zoom in/out of the imagingdevice (e.g., if the imaging device supports the zoom level required),by controlling the movement of the movable object (e.g., so as to getcloser to or farther away from the target), or by a combination of zoomin/out of the imaging device and the movement of the movable object. Aprocessor onboard the movable object may make the determination as towhich object or combination of objects to adjust. For example, if theimaging device does not support a zoom level required to maintain therequired size of the target within an image, the movable object may becontrolled to move instead of or in addition to adjusting the zoom ofthe imaging device.

In some embodiments, the adjustment may be implemented by taking intoaccount other constraints. For example, in cases where the navigationpath of the movable object is predetermined, the adjustment may beimplemented by the carrier and/or imaging device without affecting themovement of the movable object. The navigation path of the movableobject may be predetermined, for example, if a remote user is activelycontrolling the navigation of the movable object via a control terminalor if the movable object is navigating (e.g., autonomously orsemi-autonomously) according to a pre-stored navigation path.

Examples of other constraints may include maximum and/or minimum limitfor rotation angles, angular and/or linear speed, operationalparameters, and the like for the movable object, the carrier, and/or thepayload (e.g., imaging device). Such maximum and/or minimum thresholdvalues may be used to limit the range of the adjustment. For example,the angular speed of the movable object and/or the imaging device arounda certain axis may be capped by a maximum angular speed that is allowedfor the movable object, the carrier, and/or the payload (e.g., imagingdevice). As another example, the linear speed of the movable objectand/or the carrier may be capped by a maximum linear speed that isallowed for the movable object, the carrier, and/or the payload (e.g.,imaging device). As yet another example, adjustment to the focal lengthof the imaging device may be limited by the maximum and/or minimum focallength for the particular imaging device. In some embodiments, suchlimits may be predetermined and depend on the particular configurationof the movable object, the carrier, and/or the payload (e.g., imagingdevice). In some instances, such configurations may be configurable(e.g., by a manufacturer, administrator, or user).

In some embodiments, the movable object 101 can be configured to provideand the control terminal 112 can be configured to receive data such assensing data acquired by sensors onboard the movable object 101, andtracking data or information used to indicate characteristics of one ormore target tracked by the movable object 101. Examples of sensing datamay include image data acquired by an imaging device carried by themovable object 101 or other data acquired by other sensors. For example,real-time or nearly real-time video can be streamed from the movableobject 101 and/or the payload 104 (e.g., imaging device) to the controlterminal 112. The sensing data may also include data acquired by globalpositioning system (GPS) sensors, motion sensors, inertial sensors,proximity sensors, or other sensors. Examples of tracking informationmay include relative or absolute coordinates and/or size of the targetwithin one or more image frames received from the movable object,changes of the target between consecutive image frames, GPS coordinates,or other positional information of the target, and the like. In someembodiments, the tracking information may be used by the controlterminal 112 to display the target as being tracked (e.g., via agraphical tracking indicator such as a box around the target). Invarious embodiments, the data received by the control terminal mayinclude raw data (e.g., raw sensing data as acquired by the sensors)and/or processed data (e.g., tracking information as processed by one ormore processors on the movable object).

In some embodiments, the control terminal 112 can be located at alocation distant or remote from the movable object 101, carrier 102,and/or payload 104. The control terminal 112 can be disposed on oraffixed to a support platform. Alternatively, the control terminal 112can be a handheld or wearable device. For example, the control terminal112 can include a smartphone, tablet, laptop, computer, glasses, gloves,helmet, microphone, or suitable combinations thereof.

The control terminal 112 can be configured to display data received fromthe movable object 101 via a display. The displayed data may includesensing data such as images (e.g., still images and videos) acquired byan imaging device carried by the movable object 101. The displayed datamay also include tracking information that is displayed separately fromthe image data or superimposed on top of the image data. For example,the display may be configured to display the images where the target isindicated or highlighted with a tracking indicator such as a box,circle, or any other geometric shape surrounding the target beingtracked. In some embodiments, the images and the tracking indicator aredisplayed in substantially real-time as the image data and trackinginformation are received from the movable object and/or as the imagedata is acquired. In other embodiments, the display may be providedafter some delay.

The control terminal 112 can be configured to receive user input via aninput device. The input device may include a joystick, keyboard, mouse,stylus, microphone, image or motion sensor, inertial sensor, and thelike. Any suitable user input can be used to interact with the terminal,such as manually entered commands, voice control, gesture control, orposition control (e.g., via a movement, location or tilt of theterminal). For instance, the control terminal 112 may be configured toallow a user to control a state of the movable object, carrier, payload,or any component thereof by manipulating a joystick, changing anorientation or attitude of the control terminal, interacting with agraphical user interface using a keyboard, mouse, finger, or stylus, orby using any other suitable methods.

The control terminal 112 may also be configured to allow a user to entertarget information using any suitable method. In some embodiments, thecontrol terminal 112 may be configured to enable a user to directlyselect a target from one or more images (e.g., video or snapshot) thatis being displayed. For example, the user may select a target bydirectly touching the screen using a finger or stylus or selection usinga mouse or joystick. The user may draw around the target, touch thetarget in the image, or otherwise select the target. Computer vision orother techniques may be used to determine boundary of target. Otherwise,user input may define the boundary of target. One or more targets may beselected at a time. In some embodiments, the selected target isdisplayed with a selection indicator to indicate that the user hasselected the target for tracking. In some other embodiments, the controlterminal may be configured to allow a user to enter or select targettype information such as color, texture, shape, dimension, or othercharacteristics associated with a desired target. For example, the usermay type in the target type information, select such information using agraphical user interface, or use any other suitable methods. In someother embodiments, the target information may be obtained from sourcesother than the user such as a remote or local data store, othercomputing devices operatively connected to or otherwise in communicationwith the control terminal, or the like.

In some embodiments, the control terminal allows a user to selectbetween a manual tracking mode and an automatic tracking mode. When themanual tracking mode is selected, a user can specify a specific targetto track. For example, the user can manually selects a target from animage being displayed by the control terminal. The specific targetinformation associated with the selected target (e.g., coordinatesand/or size) is then provided to the movable object as initial targetinformation of the target. On the other hand, when the automatictracking mode is selected, the user does not specify a specific targetto be tracked. Rather, the user can specify descriptive informationabout the type of target to be tracked, for example, via a userinterface provided by the control terminal. The movable object can thenuse the initial target information of a specific target or target typeinformation to automatically identify the target to be tracked andsubsequently track the identified target.

In general, providing specific target information (e.g., initial targetinformation) requires more user control of the tracking of the targetand less automated processing or computation (e.g., image or targetrecognition) by a processing system onboard the movable object. On theother hand, providing target type information requires less user controlof the tracking process but more computation performed by the onboardprocessing system. The appropriate allocation of the control over thetracking process between the user and the onboard processing system maybe adjusted depending on a variety of factors such as the surroundingsof the movable object, speed or altitude of the movable object, userpreferences, and computing resources (e.g., CPU or memory) availableonboard and/or off-board the movable object, the like. For example,relatively more control may be allocated to the user when the movableobject is navigating in a relatively complex environment (e.g., withnumerous buildings or obstacles or indoor) than when the movable objectis navigating in a relatively simple environment (e.g., wide open spaceor outdoor). As another example, more control may be allocated to theuser when the movable object is at a lower altitude than when themovable object is at a higher altitude. As yet another example, morecontrol may be allocated to the movable object if the movable object isequipped with a high-speed processor adapted to perform complexcomputations relatively quickly. In some embodiments, the allocation ofcontrol over the tracking process between user and movable object may bedynamically adjusted based on the factors described herein.

The user input may be used, at least in part, to generate control datasuch as described herein. The control data may be generated by thecontrol terminal, the movable object, a third device, or any combinationthereof. For instance, the user's manipulation of a joystick or thecontrol terminal or an interaction with a graphical user interface maybe translated into predetermined control commands for changing a stateor parameter of the movable object, carrier, or payload. As anotherexample, a user's selection of a target within an image being displayedby the control terminal may be used to generate initial and/or expectedtarget information for tracking purposes such as an initial and/orexpected position and/or size of the target. Alternatively oradditionally, the control data may be generated based on informationobtained from non-user sources such as a remote or local data store,other computing devices operatively connected to the control terminal,or the like.

FIG. 2 illustrates exemplary flow 200 of image-related data amongcomponents in a tracking system, in accordance with embodiments. In someembodiments, image-related data includes raw or processed image data aswell as data extracted or derived from the image data such as trackinginformation of targets. As illustrated, raw image data can be capturedby an image sensor 204. The image sensor 204 may be configured toconvert optical signals into electronic signals. The image sensor 204may include semiconductor charge-coupled devices (CCD), active pixelsensors using complementary metal-oxide-semiconductor (CMOS) or N-typemetal-oxide-semiconductor (NMOS, Live MOS) technologies, or any othertypes of sensors. The image sensor may be coupled to a movable objectsuch as a UAV. For example, the image sensor may be part of an imagingdevice (e.g., camera) that is carried by a UAV with or without acarrier. The image sensor and/or imaging device may be configured tocapture pictures, videos, or any other image data with any suitableparameters such as width, height, aspect ratio, megapixel count,resolution or quality, and the like. For example, the imaging device maybe configured to capture high-definition or ultra-high-definition videos(e.g., 720p, 1080i, 1080p, 1440p, 2000p, 2160p, 2540p, 4000p, 4320p, andso on).

The image data captured by the image sensor 204 can be stored in a datastorage device 202. The data storage device 202 may be based onsemiconductor, magnetic, optical, or any suitable technologies and mayinclude flash memory, USB drives, memory cards, solid-state drives(SSDs), hard disk drives (HDDs), floppy disks, optical disks, magnetictapes, and the like. For example, the data storage device 202 caninclude removable storage devices that are detachably couplable to animaging device such as memory cards of any suitable formats such as PCCard, CompactFlash, SmartMedia, Memory Stick, Memory Stick Duo, MemoryStick PRO Duo, Miniature Card, Multimedia Card (MMC), Reduced SizeMultimedia Card (RS-MMC), MMCmicro Card (MMCmicro), PS2 card, SecureDigital (SD) card, SxS, Universal Flash Storage (UFS), mini SD, microSD,xD-Picture Card, Intelligent Stick (iStick), Serial Flash Module (SFM),NT Card, XQD card, and the like. The data storage device 202 can alsoinclude external hard disk drives, optical drives, tape drives, floppydrives, and other suitable storage devices that may be operativelyconnected to the imaging device.

The image data captured by the image sensor 204 can be transmitted tothe control terminal 212 by an image transmission module 206. In someembodiments, the image data may be compressed or otherwise processedbefore being transmitted by the image transmission module 206. In othercases, the image data may not be compressed or processed before beingtransmitted. The transmitted image data may be displayed on the controlterminal 212 so that a user operating the control terminal 212 can viewthe image data and/or interact with the control terminal 212 based onthe image data.

The image data captured by the image sensor 204 can be pre-processed bya pre-processing unit 208. The pre-processing unit 208 can include anyhardware, software, or a combination thereof. Examples of pre-processingunit 208 can include a field programmable gate array (FPGA). Thepre-processing unit 208 can be operatively coupled to the image sensor204 to pre-processing of the raw image data before the image data isprocessed to extract specific piece of information. Examples of tasksperformed by the pre-processing unit 208 can include re-sampling toassure the correctness of the image coordinate system, noise reduction,contrast enhancement, scale space representation, and the like.

The image data, as processed by the pre-processing unit 208, can befurther processed by a processing unit 210 that is operatively coupledto the pre-processing unit 208 (e.g., via a general purpose memorycontroller (GPMC) connection). The processing unit 210 can include oneor more ARM processors. The processing unit 210 can be configured toperform any suitable embodiments of the methods described herein.Examples of tasks performed by the processing unit 210 may includefeature extraction at any suitable level of complexity, imagesegmentation, data verification, image recognition, image registration,image matching, and the like. In some embodiments, the processing unit210 produces tracking information related to a target that is beingtracked by the movable object. The tracking information may be generatedbased on image processing by a processing unit of the movable objectand/or based on target information as provided by the control terminal.The tracking information may include, for example, the location, size,or other characteristics of a target within one or more images.

The tracking information as determined by the processing unit 210 can beprovided to the control terminal 212 via a communication module (notshown). In some embodiments, the communication module may be separatefrom the image transmission module 206 described above. In otherembodiments, the communication module may include or be included in theimage transmission module 206. Any suitable means of communication canbe used for the communication module and/or the image transmissionmodule 206, such as wired communication or wireless communication. Forexample, the communication module and/or the image transmission module206 can utilize one or more of local area networks (LAN), wide areanetworks (WAN), infrared, radio, WiFi, point-to-point (P2P) networks,telecommunication networks, cloud communication, and the like.Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. In addition to trackinginformation, the communication module can also be configured to transmitand/or receive sensing data from other sensors onboard the movableobject, positional and/or motion information determined by processingthe sensing data, predetermined control data, user commands from remotecontrol terminals, and the like.

In some embodiments, the image data, as provided by the imagetransmission module 206, may be augmented by or otherwise combined withthe tracking information, as produced by the processing unit 210, toshow a target with a tracking indicator (e.g., a circle or box aroundthe target). A user may view the augmented image data to see the targetas it is being tracked by the imaging device. The user may also interactwith the control terminal based on the augmented image data. Forexample, the user may select a different target to track from theaugmented image data (e.g., by touching an area of the screencorresponding to the different target).

In some embodiments, the tracking information can be provided to thecontrol terminal 212 in response to a demand by the control terminal212. For example, the control terminal 212 may demand such trackinginformation only when a user elects to a certain viewing mode (e.g., apreview mode) where the target being tracked is highlighted or otherwiseindicated. Alternatively or additionally, the tracking information maybe provided to the control terminal 212 without any demand by thecontrol terminal 212. For example, the tracking information may bepushed to the control terminal 212 on a periodic basis (e.g., every 0.1second, 0.2 second, 0.5 second, 1 second, or 2 second).

FIG. 3 illustrates exemplary flow 300 of control data among componentsin a tracking system, in accordance with embodiments. As discussedabove, control data can include target information used by a movableobject (e.g., a UAV) to track a target, as well as other data forcontrolling various aspects of the movable object or a componentthereof. The control data can be generated by a user 302 interactingwith a control terminal 304. The generated control data can includespecific target information (e.g., initial target information) 306 andtarget type information 308.

Specific target information 306 can include characteristics about aspecific target such as coordinates (e.g., pixel coordinates), size, andthe like. In some embodiments, specific target information 306 can begenerated when a user selects or specifies a specific target to trackvia a user interface provided by the control terminal 304. For example,the control terminal may allow a user to select between a manualtracking mode and an automatic tracking mode. When the manual trackingmode is selected, a user can specify a specific target to track (e.g.,by selecting a target from one or more images being displayed). Based onthe user selection, specific target information may be generated.

Target type information 308 can include information describing a type oftargets to be tracked rather than information about a specific target.Such target type information may include various target characteristicssuch as color, texture, pattern, size, shape, dimension, and the like.In some embodiments, the target information (including specific targetinformation and target type information) may be generated at least inpart on user input at the control terminal 304. For example, the controlterminal may allow a user to select between a manual tracking mode andan automatic tracking mode. When the automatic tracking mode isselected, a user can enter or select target type information.Additionally or alternatively, the target information may be generatedbased on data from other sources. For example, target type informationmay be derived based on previous images and/or extracted from local orremote data stores. In some embodiments, predefined target types orpatterns may be presented to the user for selection. In some otherembodiments, the predefined target types or patterns may be providedautomatically by default to the movable object without userintervention.

Target information can optionally include expected target informationsuch as described herein. The expected target information may or may notoverlap with the initial target information.

The target information (including specific target information 306 andtarget type information 308) can be provided to a processing system 310,for example, via a communication module (not shown). The processingsystem 310 may be onboard a movable object (e.g., UAV). The processingsystem 310 may include a data processing unit 312 and a command controlunit 314. The data processing unit 312 may be configured to perform anyembodiments of the methods described herein. For instance, the dataprocessing unit 312 can be configured to identify target based on targetinformation (e.g., including specific target information 306 and/ortarget type information 308), determine deviation from the targetinformation, and the like. The data processing unit 312 may include apre-processing unit and/or a processing unit such as similar to thepre-processing unit 208 and processing unit 210 respectively describedin FIG. 2. For example, the data processing unit 312 may include an FPGAand/or one or more ARM processors.

The data processing unit 312 may be operatively coupled to a commandcontrol module 314 configured to control a state of the movable object.The command control module 314 may be configured to perform anyembodiments of the methods described herein. For instance, the commandcontrol module 314 can be configured to generate control commands orsignals 316 for controlling a component of the movable object so as tosubstantially track the target based on the results from the dataprocessing unit 312.

The control commands 316 can include commands for the propulsionmechanisms of the movable object to adjust the spatial disposition,velocity, and/or acceleration of the movable object with respect to upto six degrees of freedom in order to correct a detected deviation ofthe target with respect to its position and/or size in one or more ofthe images. The control commands 316 can also include commands foradjusting the state of a carrier so as to adjust the spatialdisposition, velocity, and/or acceleration of a payload (e.g., imagingdevice) carried by the movable object via the carrier.

The control commands 316 can also include commands for adjusting thestate of a carrier so as to adjust the spatial disposition, velocity,and/or acceleration of a payload (e.g., imaging device) carried by themovable object via the carrier.

The control commands 316 can also include commands for adjusting one ormore operating parameters of the payload such as taking still or movingpictures, zooming in or out, turning on or off, switching imaging modes,change image resolution, changing focus, changing depth of field,changing exposure time, changing speed of lens, changing viewing angleor field of view, or the like.

Alternatively or additionally, any of the above control commands can beprovided directly from the control terminal 304 to the processing system310. For example, the user may use the control terminal 304 to controlthe zoom in/out of the imaging device while the processing system 310generates the control commands for the movable object and/or thecarrier. As another example, the user may directly control the movableobject while the processing system 310 generates the control commandsfor the movable object and/or the imaging device. This may allow a userto focus on controlling the navigation of the movable object withouthaving to worry about tracking the target, which is performedautomatically by the processing system 310.

In various embodiments, the allocation of tracking control between theuser and the automatic tracking system can vary depending on a varietyof factors such as the surroundings of the movable object, speed oraltitude of the movable object, user preferences, and computingresources (e.g., CPU or memory) available onboard and/or off-board themovable object, the like. For example, relatively more control may beallocated to the user when the movable object is navigating in arelatively complex environment (e.g., with numerous buildings orobstacles or indoor) than when the movable object is navigating in arelatively simple environment (e.g., wide open space or outdoor). Asanother example, more control may be allocated to the user when themovable object is at a lower altitude than when the movable object is ata higher altitude. As yet another example, more control may be allocatedto the movable object if the movable object is equipped with ahigh-speed processor adapted to perform complex computations relativelyquickly. In some embodiments, the allocation of control over thetracking process between user and movable object may be dynamicallyadjusted based on the factors described herein.

According to an aspect of the present invention, methods and systems areprovided for tracking a target by an imaging device coupled to a movableobject so as to substantially maintain an expected position and/or sizeof the target within one or more images captured by the imaging device.

FIG. 4 illustrates an exemplary process 400 for implementing targettracking, in accordance with embodiments. Aspects of the process 400 maybe performed by one or more processors onboard and/or off-board amovable object as described herein such as a UAV. Some or all aspects ofthe process 400 (or any other processes described herein, or variationsand/or combinations thereof) may be performed under the control of oneor more computer/control systems configured with executable instructionsand may be implemented as code (e.g., executable instructions, one ormore computer programs or one or more applications) executingcollectively on one or more processors, by hardware or combinationsthereof. The code may be stored on a computer-readable storage medium,for example, in the form of a computer program comprising a plurality ofinstructions executable by one or more processors. The computer-readablestorage medium may be non-transitory. The order in which the operationsare described is not intended to be construed as a limitation, and anynumber of the described operations may be combined in any order and/orin parallel to implement the processes.

The process 400 includes obtaining 402 target information for one ormore targets. The target information can be received from a controlterminal such as described herein. Additionally or alternatively, thetarget information may be obtained from a component (e.g., memory)onboard a movable object (e.g., UAV) or a device remote from the movableobject such as another movable object (e.g., another UAV), a server, orthe like. In some cases, the target information about a target may beprovided by the target itself.

In various embodiments, the target information may include initialtarget information specific to a particular known target or target typeinformation about potentially unknown target(s). Specific targetinformation include coordinates (e.g., pixel coordinates), size,location, and other information about a target within one or moreimages. Specific target information may be generated based on userinteraction with existing image data such as described herein. Forexample, specific target information may be generated when a userselects a particular object as target from one or more images displayedto the user. The specific target information may include the initialposition and/or size of the target as the target is selected by a remoteuser from within one or more images.

Target information may also include target type information such ascolor, texture, dimension, size, location, and/or any othercharacteristics about a type or group of potentially unknown orunidentified targets. Target type information may be specificallyentered by a user. Alternatively, the user may select a pre-existingtarget pattern or type (e.g., a black object or a round object with aradius greater or less than a certain value). In some embodiments, theuser may provide target type information by selecting one or moretargets from within one or more images. Features or characteristics ofthe selected targets may then be extracted and/or generalized to producethe target type information, which may be used to identify other targetswith similar features or characteristics. In various embodiments, suchfeature extraction may be performed by a control terminal, a processingunit on the movable object, or third device.

Target information (including specific target information and targettype information) may optionally include expected characteristics atwhich the target, if identified, should maintain while the target istracked. For example, the target information may include an expectedposition of the target as expressed by absolute or relative coordinateswithin an image. The tracking system may be configured to track thetarget such that the target is kept, within a predefined degree oftolerance, at substantially the same expected position over time.Alternatively or additionally, the target information may include anexpected size of the target (e.g., as expressed by a number of pixelsoccupied by the target). The tracking system may be configured to trackthe target such that the target is kept, within a predefined degree oftolerance, at substantially the same expected size. Such expected targetinformation may be the same or different from the initial targetinformation for a specific target. The initial target information istypically used to identify a target. Once the target has beenidentified, the expected target information may be used to detect anydeviation from the expected target information so that such deviationcan be corrected. In some cases, the target information may also includeother values such as a time value or expiration time indicating a periodof time during which the target should be tracked, if identified, a flagindicating whether the target information includes specific targetinformation or target type information, and the like.

In some embodiments, the expected target information at which the targetis to maintain may be provided by the control terminal. For example, theexpected target information may be generated based on user input to thecontrol terminal or based on a configuration file or data store local orremote to the control terminal. In some other embodiments, the expectedtarget information may be provided by the movable object. For example,the movable object may be configured, by default, to keep a target atsubstantially the center of an image, or at around particularcoordinates of the image. As another example, the movable object may beconfigured, by default, to keep the target as captured by the imagingdevice, at a particular size. In yet some other embodiments, theexpected target information may be provided by some other objects ordevice external to the movable object.

Once the target information is received, the process 400 includesidentifying 404 a target based on the target information, e.g., based ontarget type information. Any suitable image recognition oridentification techniques may be used including approaches based onCAD-like object models (e.g., edge detection, primal sketch, Marr, Mohanand Nevatia, Lowe, or Olivier Faugeras), appearance-based methods (e.g.,using edge matching, divide-and-conquer search, greyscale matching,gradient matching, historgrams of receptive field responses, or largemodel bases), feature-based methods (e.g., using interpretation trees,hypothesizing and testing, pose consistency, pose clustering,invariance, geometric hashing, Scale-invariant feature transform (SIFT),or Speeded Up Robust Features (SURF)), genetic algorithms, and the like.

After the target has been identified, subsequent images captured by theimaging device may be monitored to detect 406 a deviation of the targetfrom certain expected characteristics that the target should maintainsuch as expected position and/or size. As discussed above, the expectedtarget information may be supplied by a control terminal (e.g., based onuser input), a configuration file or memory associated with the movableobject, or from some other sources. The expected target information mayor may not be the same as the initial target information. The deviationof the target may be achieved by comparing the target's respectiveposition, size and/or any other suitable characteristics in one or moreimages with expected characteristics. Any suitable image recognition oridentification techniques such as discussed herein may be used.

In some embodiments, the expected target information is consideredsubstantially maintained when the detected deviation falls withincertain predefined tolerance values. In such cases, the deviation may beconsidered negligible and no corrective adjustment is required. Onlywhen the deviation exceeds the predefined tolerance value is correctiveadjustment required. For example, when the current position of thetarget is within a predetermined number of pixels from the expectedcoordinates, the deviation may be considered negligible and hence nocorrective adjustment is required. Similarly, when the current size ofthe target is within a predetermined number of pixels from the expectedsize, the deviation may be considered negligible. The predefined degreeof tolerance may be defined by system parameters, configured by usersoperating the control terminal, or otherwise defined.

In order to correct the deviation and maintain substantially theexpected characteristics of the target, control signals or commands canbe generated 408 for adjusting the movable, carrier, and/or imagingdevice. In some embodiments, deviations in the position of the targetcan be corrected via adjustment to the attitude of the movable objectand/or the imaging device (via the carrier), such as discussed infurther detail in FIG. 6. For example, such adjustment may involvechanging angular velocity of the movable object and/or imaging devicearound one or more rotational axes. Deviations in the size of the targetcan be corrected via adjustment to the position of the movable objectand/or to the operational parameters of the imagining device, such asdiscussed in further detail in FIG. 7. For example, such adjustment mayinvolve changing linear velocity of the movable object along an axis.Alternatively or additionally, the adjustment may involve changing thezoom, focus, or other characteristics associated with the imagingdevice.

In some embodiments, the adjustment may be limited by constraintsimposed by system configuration, by the user operating a controlterminal, or by other entities. Examples of such constraints may includemaximum and/or minimum limit for rotation angles, angular and/or linearspeed, and the like for the movable object (e.g., the propulsion systemthereof), the carrier (e.g., an actuation member thereof), the imagingdevice, or the like. Such threshold values may be used to limit therange of the adjustment. For example, an adjustment involving theangular speed of the movable object and/or the imaging device relativeto the movable object (via a carrier) around a certain axis may becapped by a maximum angular speed that is allowed for the movable objectand/or the carrier. As another example, an adjustment involving thelinear speed of the movable object and/or the imaging device relative tothe movable object (via a carrier) may be capped by a maximum linearspeed that is allowed for the movable object and/or the carrier. In someembodiments, such limits may be predetermined and depend on theparticular configuration of the movable object, carrier, and/or theimaging device. In some embodiments, the limits may be configurable(e.g., by a manufacturer, administrator, or user).

In some embodiments, warning indicators may be provided when theadjustment is modified according to the constraints described above(e.g., when the angular speed of the movable object and/or the carrieraround a certain axis is capped by a maximum angular speed that isallowed for the movable object and/or the carrier). Examples of suchwarning indicators may include textual, audio (e.g., siren or beepingsound), visual (e.g., certain color of light or flashing light), kinetic(e.g., vibration or movement), any other suitable types of signals. Suchwarning indicators may be provided directly by the movable object or acomponent thereof. Alternatively or additionally, warning indicators maybe provided by the control terminal (e.g., via the display). In thelatter case, the control terminal may provide the warning indicatorsbased on signals from the movable object.

In some embodiments, the adjustment may be performed in substantiallyreal time as the movable object is executing user-provided navigationcommands or a predetermined flight path, and/or as the imaging device iscapturing one or more images. In some embodiments, the adjustment may begenerated based on other information such as sensing data acquired byone or more sensors onboard the movable object (e.g., proximity sensor,or GPS sensor). For example, position information of the target beingtracked may be obtained by a proximity sensor and/or provided by thetarget itself (e.g., GPS location). Such position information may beused, in addition to the detected deviation, to generate the adjustmentcontrol signals to track the target such as described herein.

In various embodiments, detection 406 of deviation from expected targetinformation and/or generation 408 of control commands to correct thedetected deviation may be repeated for a predefined or indefinite periodof time. In some embodiments, such deviation detection and/or controlcommand generation may be performed at certain intervals (e.g., every0.01 second, 0.1 second, 0.2 second, 0.5 second, or 1 second).

FIG. 5 illustrates an exemplary configuration 500 of a movable object,carrier, and payload, in accordance with embodiments. The configuration500 is used to illustrate exemplary types of adjustment to the movableobject 502 and/or payload 506 that may be used to track a target. Themovable object 502 and the payload 506 can include any embodimentsdiscussed herein. For example, the movable object 502 can include a UAVand the payload 506 can include an imaging device.

The movable object 502 may be capable of rotating around up to threeorthogonal axes, such as X₁ (pitch) 510, Y₁ (yaw) 508 and Z₁ (roll) 512axes. The rotations around the three axes can be referred to as thepitch rotation 522, yaw rotation 520, and roll rotation 524,respectively. The angular velocities of the movable object 502 aroundthe X₁, Y₁, and Z₁ axes can be expressed as ω_(X1), ω_(Y1), and ω_(Z1),respectively. The movable object 502 may also be capable oftranslational movements 528, 526, and 530 along the X₁, Y₁, and Z₁ axes,respectively. The linear velocities of the movable object 502 along theX₁, Y₁, and Z₁ axes can be expressed as V_(X1), V_(Y1), and V_(Z1),respectively.

In the exemplary configuration, the payload 506 is coupled to themovable object 502 via a carrier 504. The carrier 504 may be capable ofcausing the payload 506 to move relative to the movable object aroundand/or along up to three orthogonal axes, X₂ (pitch) 516, Y₂ (yaw) 514and Z₂ (roll) 518. The X₂, Y₂, and Z₂ axes may be respectively parallelto the X₁, Y₁, and Z₁ axes. In some embodiments, where the payload is animaging device including an optical module 507, the roll axis Z₂ 518 maybe substantially parallel to an optical path or optical axis for theoptical module 507. The optical module 507 may be optically coupled toan image sensor such as described herein to capture images. The carrier504 may cause the carrier 506 to rotate around up to three orthogonalaxes, X₂ (pitch) 516, Y₂ (yaw) 514 and Z₂ (roll) 518 based on controlsignals provided to actuators associated with the carrier such aselectric motors. The rotations around the three axes can be referred toas the pitch rotation 534, yaw rotation 532, and roll rotation 536,respectively. The angular velocities of the payload 506 around the X₂,Y₂, and Z₂ axes can be expressed as ω_(X2), ω_(Y2), and ω_(Z2),respectively. The carrier 504 may also cause the payload 506 to engagein translational movements 540, 538, and 542, along the X₂, Y₂, and Z₂axes relative to the movable object 502. The linear velocity of thepayload 506 along the X₂, Y₂, and Z₂ axes can be expressed as V_(X2),V_(Y2), and V_(Z2), respectively.

In some embodiments, the carrier 504 may only permit the payload 506 tomove around and/or along a subset of the three axes X₂, Y₂, and Z₂relative to the movable object 502. For instance, the carrier 504 mayonly permit the payload 506 rotate around X₂, Y₂, Z₂, or any combinationthereof, without allowing the payload 506 to move along any of the axes.For example, the carrier 504 may permit the payload 506 to rotate onlyaround one of the X₂, Y₂, and Z₂ axes. The carrier 504 may permit thepayload 506 to rotate only around two of the X₂, Y₂, and Z₂ axes. Thecarrier 504 may permit the payload 506 to rotate around all three of theX₂, Y₂, and Z₂ axes.

In some other cases, the carrier 504 may only permit the payload 506move along X₂, Y₂, or Z₂ axis, or any combination thereof, withoutallowing the payload 506 to rotate around any of the axes. For example,the carrier 504 may permit the payload 506 to move along only one of theX₂, Y₂, and Z₂ axes. The carrier 504 may permit the payload 506 to movealong only two of the X₂, Y₂, and Z₂ axes. The carrier 504 may permitthe payload 506 to move along only all three of the X₂, Y₂, and Z₂ axes.

In yet some other embodiments, the carrier 504 may allow the payload 506perform both rotational and translational movement relative to themovable object. For example, the carrier 504 may be configured to allowthe payload 506 to move along and/or rotate around one, two, or three ofthe X₂, Y₂, and Z₂ axes.

In some other embodiments, the payload 506 may be coupled to the movableobject 502 directly without a carrier 504, or the carrier 504 may notpermit the payload 506 to move relative to the movable object 502. Insuch cases, the attitude, position and/or orientation of the payload 506is fixed relative to the movable object 502.

In various embodiments, adjustment to attitude, orientation, and/orposition of the payload 506 may be achieved, collectively orindividually, via suitable adjustment to the movable object 502, thecarrier 504, and/or the payload 506. For example, a rotation of 60degrees around a given axis (e.g., yaw axis) for the payload may beachieved by a 60-degree rotation by the movable object alone, a60-degree rotation by the payload relative to the movable object aseffectuated by the carrier, or a combination of 40-degree rotation bythe movable object and a 20-degree rotation by the payload relative tothe movable object.

Similarly, a translational movement for the payload may be achieved,collectively or individually, via adjustment to the movable object 502and the carrier 504. The desired adjustment may, additionally oralternatively, be achieved by adjustment to the operational parametersof the payload. Such operational parameters of the payload may include,for example, a zoom in/out level or a focal length of an imaging device.

FIG. 6 illustrates an exemplary tracking method for maintaining anexpected position of a target, in accordance with embodiments. Anexemplary image 600 is shown such as captured by an imaging devicecarried by a movable object. Assume that the image has a width of Wpixels and a height of H pixels (where W and H are positive integers). Aposition within the image can be defined by a pair of coordinates alonga horizontal axis 601 (along the width of the image) and a vertical axis603 (along the height of the image), where the upper left corner ofimage has coordinates (0, 0) and the lower right corner of the image hascoordinates (W, H).

Assume that a target, as captured in the image 600, is located atposition P (u, v) 602, and the expected position of the target is P₀(u₀, v₀) 604 that is different from P 602. In some embodiments, theexpected position of the target P₀ (u₀, v0) may be near the center ofthe image, such that u₀=W/2, and/or v₀=H/2. In other embodiment, theexpected position of the target may be located anywhere else within theimage (e.g., off-center). In various embodiments, the expected positionof the target may or may not be the same as the initial position of thetarget (e.g., as provided by the control terminal). Assuming that thecurrent position P is deviated from the expected position P₀ such thatthe deviation exceeds a predetermined threshold (such as expressed by aΔx from u₀, and a Δy from v₀), then an adjustment is required to bringthe target position from P to close to the expected position P₀.

In some embodiments, the deviation from the expected target position canbe used to derive one or more angular velocities for rotating the fieldof view of the imaging device around one or more axes. For example,deviation along the horizontal axis 601 of the image (e.g., between uand u₀) may be used to derive an angular velocity ω_(Y) 610 for rotatingthe field of view of the imaging device around the Y (yaw) axis 606, asfollows:ω_(Y)=α*(u−u ₀), where α∈

(real numbers)  (1)

The rotation around the Y axis for the field of view of an imagingdevice may be achieved by a rotation of the movable object, a rotationof the payload (via a carrier) relative to the movable object, or acombination of both. In some embodiments, adjustment to the payload maybe selected when adjustment to the movable object is infeasible orotherwise undesirable, for example, when the navigation path of themovable object is predetermined. In the equation (1), α is a constantthat may be predefined and/or calibrated based on the configuration ofthe movable object (e.g., when the rotation is achieved by the movableobject), the configuration of the carrier (e.g., when the rotation isachieved by the carrier), or both (e.g., when the rotation is achievedby a combination of the movable object and the carrier). In someembodiments, α is greater than zero (α>0). In other embodiments, α maybe no greater than zero (α≤0). In some embodiments, α can be used to mapa calculated pixel value to a corresponding control lever amount orsensitivity for controlling the angular velocity around a certain axis(e.g., yaw axis). In general, the control lever may be used to controlthe angular or linear movement of a controllable object (e.g., UAV orcarrier). Greater control lever amount corresponds to greatersensitivity and greater speed (for angular or linear movement). In someembodiments, the control lever amount or a range thereof may bedetermined by configuration parameters of the flight control system fora UAV or configuration parameters of a control system for a carrier. Theupper and lower bounds of the range of the control lever amount mayinclude any arbitrary numbers. For example, the range of the controllever amount may be (1000, −1000) for one flight control system and(−1000, 1000) for another flight control system.

For instance, assume that the images have a width of W=1024 pixels and aheight of H=768 pixels. Thus, the size of the images is 1024*768.Further assume that the expected position of the target has a u₀=512.Thus, (u−u₀)∈(−512, 512). Assume that the range of the control leveramount around the yaw axis is (−1000, 1000), then the maximum controllever amount or maximum sensitivity is 1000 and α=1000/512. Thus, thevalue of a can be affected by image resolution or size provided by theimaging device, range of the control lever amount (e.g., around acertain rotation axis), the maximum control lever amount or maximumsensitivity, and/or other factors.

For instance, when the rotation is achieved by rotation of the movableobject, the Y axis 606 of FIG. 6 corresponds to the Y₁ axis 508 for themovable object as illustrated in FIG. 5 and the overall angular velocityof the field of view ω_(Y) is expressed as the angular velocity ω_(Y1)for the movable object:ω_(Y)=ω_(Y1)=α₁*(u−u ₀), where α₁∈

  (2)

In the equation (2), α₁ is a constant that is defined based on theconfiguration of the movable object. In some embodiments, α₁ is greaterthan zero (α₁>0). The α₁ can be defined similar to the α discussedabove. For example, the value of α₁ may be defined based on imageresolution or size and/or range of control lever amount for the movableobject (e.g., around the yaw axis).

Similarly, when the rotation is achieved by the rotation of the payloadrelative to the movable object (e.g., via the carrier), the Y axis 606of FIG. 6 corresponds to the Y₂ axis 514 for the payload as illustratedin FIG. 5 and the overall angular velocity of the field of view ω_(Y) isexpressed as the angular velocity ω_(Y2) for the payload relative to themovable object:ω_(Y)=ω_(Y2)=α₂*(u−u ₀), where α₂∈

  (3)

In the equation (3), α₂ is a constant that is defined based on theconfiguration of the carrier and/or payload. In some embodiments, α₂ isgreater than zero (α₂>0). The α₂ can be defined similar to the αdiscussed above. For example, the value of α₂ may be defined based onimage resolution or size and/or range of control lever amount for thecarrier (e.g., around the yaw axis).

In general, the angular velocity of the field of view around the Y (yaw)axis 606 can be expressed as a combination of the angular velocityω_(Y1) for the movable object and the angular velocity ω_(Y2) for thepayload relative to the movable object, such as the following:ω_(Y)=ω_(Y1)+ω_(Y2)  (4)

In the equation (4), either ω_(Y1) or ω_(Y2) may be zero.

As illustrated herein, the direction of the rotation around the Y (yaw)axis may depend on the sign of u−u₀. For instance, if the expectedposition is located to the right of the actual position (as illustratedin FIG. 6), then u−u₀<0, and the field of view needs to rotate in acounter-clockwise fashion around the yaw axis 606 (e.g., pan left) inorder to bring the target to the expected position. On the other hand,if the expected position is located to the left of the actual position,then u−u₀>0, and the field of view needs to rotate in a clockwisefashion around the yaw axis 606 (e.g., pan right) in order to bring thetarget to the expected position.

As illustrated herein, the speed of rotation (e.g., absolute value ofthe angular velocity) around a given axis (e.g., the Y (yaw) axis) maydepend on the distance between the expected and the actual position ofthe target along the axis (i.e., |u−u₀|). The further the distance is,the greater the speed of rotation. Likewise, the closer the distance is,the slower the speed of rotation. When the expected position coincideswith the position of the target along the axis (e.g., u=u₀), then thespeed of rotation around the axis is zero and the rotation stops.

The method for adjusting the deviation from the expected target positionand the actual target position along the horizontal axis 601, asdiscussed above, can be applied in a similar fashion to correct thedeviation of the target along a different axis 603. For example,deviation along the vertical axis 603 of the image (e.g., between v andv₀) may be used to derive an angular velocity ω_(X) 612 for the field ofview of the imaging device around the X (pitch) axis 608, as follows:ω_(X)=β*(v−v ₀), where β∈

  (5)

The rotation around the X axis for the field of view of an imagingdevice may be achieved by a rotation of the movable object, a rotationof the payload (via a carrier) relative to the movable object, or acombination of both. Hence, in the equation (5), β is a constant thatmay be predefined and/or calibrated based on the configuration of themovable object (e.g., when the rotation is achieved by the movableobject), the configuration of the carrier (e.g., when the rotation isachieved by the carrier), or both (e.g., when the rotation is achievedby a combination of the movable object and the carrier). In someembodiments, β is greater than zero (β>0). In other embodiments, β maybe no greater than zero (β≤0). In some embodiments, can be used to map acalculated pixel value to a corresponding control lever amount forcontrolling the angular velocity around a certain axis (e.g., pitchaxis). In general, the control lever may be used to control the angularor linear movement of a controllable object (e.g., UAV or carrier).Greater control lever amount corresponds to greater sensitivity andgreater speed (for angular or linear movement). In some embodiments, thecontrol lever amount or a range thereof may be determined byconfiguration parameters of the flight control system for a UAV orconfiguration parameters of a carrier control system for a carrier. Theupper and lower bounds of the range of the control lever amount mayinclude any arbitrary numbers. For example, the range of the controllever amount may be (1000, −1000) for one control system (e.g., flightcontrol system or carrier control system) and (−1000, 1000) for anothercontrol system.

For instance, assume that the images have a width of W=1024 pixels and aheight of H=768 pixels. Thus, the size of the images is 1024*768.Further assume that the expected position of the target has a v₀=384.Thus, (v−v₀)∈(−384, 384). Assume that the range of the control leveramount around the pitch axis is (−1000, 1000), then the maximum controllever amount or maximum sensitivity is 1000 and β=1000/384. Thus, thevalue of β can be affected by image resolution or size provided by theimaging device, range of the control lever amount (e.g., around acertain rotation axis), the maximum control lever amount or maximumsensitivity, and/or other factors.

For instance, when the rotation is achieved by rotation of the movableobject, the X axis 608 of FIG. 6 corresponds to the X₁ axis 510 for themovable object as illustrated in FIG. 5 and the angular velocity of thefield of view ω_(X) is expressed as the angular velocity ω_(X1) for themovable object:ω_(X)=ω_(X1)=β₁*(v−v ₀), where β₁∈

  (6)

In the equation (6), β₁ is a constant that is defined based on theconfiguration of the movable object. In some embodiments, β₁ is greaterthan zero (β₁>0). The β₁ can be defined similar to the β discussedabove. For example, the value of β₁ may be defined based on imageresolution or size and/or range of control lever amount for the movableobject (e.g., around the pitch axis).

Similarly, when the rotation is achieved by the rotation of the payloadrelative to the movable object (e.g., via the carrier), the X axis 608of FIG. 6 corresponds to the X₂ axis 516 for the payload as illustratedin FIG. 5 and the angular velocity of the field of view ω_(X) isexpressed as the angular velocity ω_(X2) for the payload relative to themovable object:ω_(X)=ω_(X2)=β₂*(v−v ₀), where β₂∈

  (6)

In the equation (6), β₂ is a constant that is defined based on theconfiguration of the carrier and/or payload. In some embodiments, β₂ isgreater than zero (β₂>0). The β₂ can be defined similar to the βdiscussed above. For example, the value of β₂ may be defined based onimage resolution or size and/or range of control lever amount for themovable object (e.g., around the pitch axis).

In general, the angular velocity of the field of view around the X(pitch) axis 608 can be expressed as a combination of the angularvelocity ω_(X1) for the movable object and the angular velocity ω_(X2)for the payload relative to the movable object, such as the following:ω_(X)=ω_(X1)+ω_(X2)  (7)

In the equation (7), either ω_(X1) or ω_(X2) may be zero.

As illustrated herein, the direction of the rotation around the X (yaw)axis may depend on the sign of v−v₀. For instance, if the expectedposition is located above of the actual position (as illustrated in FIG.6), then v−v₀>0, and the field of view needs to rotate in a clockwisefashion around the pitch axis 608 (e.g., pitch down) in order to bringthe target to the expected position. On the other hand, if the expectedposition is located to below the actual position, then v−v₀<0, and thefield of view needs to rotate in a counter-clockwise fashion around thepitch axis 608 (e.g., pitch up) in order to bring the target to theexpected position.

As illustrated herein, the speed of rotation (e.g., absolute value ofthe angular velocity) depends on the distance between the expected andthe actual position of the target (i.e., |v−v₀|) along a give axis(e.g., the X (pitch) axis). The further the distance is, the greater thespeed of rotation. The closer the distance is, the slower the speed ofrotation. When the expected position coincides with the position of thetarget (e.g., v=v₀), then the speed of rotation is zero and the rotationstops.

In some embodiments, the values of the angular velocities as calculatedabove may be constrained or otherwise modified by various constraints ofthe system. Such constraints may include the maximum and/or minimumspeed that may be achieved by the movable object and/or the imagingdevice, the range of control lever amount or the maximum control leveramount or maximum sensitivity of the control system for the movableobject and/or the carrier, and the like. For example, the rotation speedmay be the minimum of the calculated rotation speed and the maximumspeed allowed.

In some embodiments, warning indicators may be provided when thecalculated angular velocities need to be modified according to theconstraints described herein. Examples of such warning indicators mayinclude textual, audio (e.g., siren or beeping sound), visual (e.g.,certain color of light or flashing light), mechanical, any othersuitable types of signals. Such warning indicators may be provideddirectly by the movable object, carrier, payload, or a componentthereof. Alternatively or additionally, warning indicators may beprovided by the control terminal (e.g., via the display). In the lattercase, the control terminal may provide the warning indicators based onsignals from the movable object.

FIG. 7 illustrates an exemplary tracking method for maintaining anexpected size of a target, in accordance with embodiments. An exemplaryimage 700 is shown such as captured by an imaging device carried by amovable object. Assume that a target 702 is captured by the image 700.The actual size of the target within the image can be s pixels (such ascalculated as the product of the width of the target and the height ofthe target). The expected target size S may be smaller (e.g., theexpected target may be represented by 704 and S=s₀) or larger (e.g., theexpected target may be represented by 705 and S=s₁) than the actual sizes. The expected size of the target may or may not be the same as theinitial size of the target (e.g., as provided by the control terminal).Assuming that the current size s is deviated from the expected size s₀or s₁ such that the deviation exceeds a predetermined threshold (such asa predefined Δs pixels), then an adjustment is required to bring thetarget size close to the expected size s₀ or s₁.

Although display area of the image and target is shown as rectangles, itis for illustrative purposes only and not intended to be limiting.Rather, the display area of the image and/or target may be of anysuitable shapes in various embodiments such as circles, ovals, polygons,and the like. Likewise, although the areas discussed herein areexpressed in pixels, these are for illustrative purposes only and notintended to be limiting. In other embodiments, the areas may beexpressed in any suitable units such as megapixels, mm², cm², inch², andthe like.

In some embodiments, the deviation from the expected target size can beused to derive one or more linear velocities for the movable objectand/or imaging device along one or more axes. For example, deviation inthe target size between actual target size s and the expected targetsize S (e.g., S=s₀ or s₁) can be used to determine a linear velocity Vfor moving the movable object along a Z (roll) axis 710, as follows:V=δ*(1−s/S), where δ∈

  (8)

In the equation (8), δ is a constant that is defined based on theconfiguration of the movable object or any suitable controllable object(e.g., carrier) that may cause the field of view to move toward and/oraway from the target. In some embodiments, δ is greater than zero (δ>0).In other embodiments, δ may be no greater than zero (δ≤0). In someembodiments, δ can be used to map a calculated pixel value to acorresponding control lever amount or sensitivity for controlling thelinear velocity.

In general, V represents the velocity of the movable object toward oraway from the target. The velocity vector points from the UAV to thetarget. If the actual size s of the target is smaller than the expectedsize S, then V>0 and the movable object moves towards the target so asto increase the size of the target as captured in the images. On theother hand, if the actual size s of the target is larger than theexpected size S, then V<0 and the movable object moves away from thetarget so as to reduce the size of the target as captured in the images.

For instance, assume that the images have a width of W=1024 pixels and aheight of H=768 pixels. Thus, the size of the images is 1024*768. Assumethat the range of the control lever amount for controlling the linearvelocity is (−1000, 1000). In an exemplary embodiment, δ=−1000 whens/S=3 and δ=1000 when s/S=⅓.

In some embodiments, the values of the velocities as calculated abovemay be constrained or otherwise modified by various constraints of thesystem. Such constraints may include the maximum and/or minimum speedthat may be achieved by the movable object and/or the imaging device,the maximum sensitivity of the control system for the movable objectand/or the carrier, and the like. For example, the speed for the movableobject may be the minimum of the calculated speed and the maximum speedallowed.

Alternatively or additionally, the deviation between the actual targetsize and the expected target size can be used to derive adjustment tothe operational parameters of the imaging device such as a zoom level orfocal length in order to correct the deviation. Such adjustment to theimaging device may be necessary when adjustment to the movable object isinfeasible or otherwise undesirable, for example, when the navigationpath of the movable object is predetermined. An exemplary focal lengthadjustment F can be expressed as:F=γ*(1−s/S), where γ∈

  (9)

Where γ is a constant that is defined based on the configuration of theimaging device. In some embodiments, γ is greater than zero (γ>0). Inother embodiments, γ is no greater than zero (γ≤0). The value of γ maybe defined based on the types of lenses and/or imaging devices.

If the actual size s of the target is smaller than the expected size S,then F>0 and the focal length increases by |F| so as to increase thesize of the target as captured in the images. On the other hand, if theactual size s of the target is larger than the expected size S, then F<0and the focal length decreases by |F| so as to reduce the size of thetarget as captured in the images. For example, in an embodiment, γ=10.This means that, for example, when the actual size of the target isdouble the size of the expected size S, the focal length should bedecreased by 10 mm accordingly (i.e., F=10*(1−2/1)=−10) and vice versa.

In some embodiments, the adjustment to the operational parameters of theimaging device such as focal length may be constrained or otherwisemodified by various constraints of the system. Such constraints mayinclude, for example, the maximum and/or minimum focal lengths that maybe achieved by the imaging device. As an example, assume the focallength range is (20 mm, 58 mm). Further assume that the initial focallength is 40 mm. Then when s>S, the focal length should be decreasedaccording to equation (9); and when s<S, the focal length should beincreased according to equation (9). However, such adjustment is limitedby the lower and upper bounds of the focal length range (e.g., 20 mm to58 mm). In other words, the post-adjustment focal length should be noless than the minimum focal length (e.g., 20 mm) and no more than themaximum focal length (e.g., 58 mm).

As discussed above in FIG. 6, warning indicators may be provided whenthe calculated adjustment (e.g., linear velocity of movable object orfocal length) is modified according to the constraints described herein.Examples of such warning indicators may include textual, audio (e.g.,siren or beeping sound), visual (e.g., certain color of light orflashing light), mechanical, any other suitable types of signals. Suchwarning indicators may be provided directly by the movable object,carrier, payload, or a component thereof. Alternatively or additionally,warning indicators may be provided by the control terminal (e.g., viathe display). In the latter case, the control terminal may provide thewarning indicators based on signals from the movable object.

According to another aspect of the present invention, methods andsystems are provided for tracking a target by an imaging device coupledto a movable object while the movable object is controlled to navigateby navigation commands.

FIG. 8 illustrates another exemplary process 800 for implementing targettracking, in accordance with embodiments. Aspects of the process 800 maybe performed by one or more processors onboard a movable object asdescribed herein such as a UAV.

The process 800 includes receiving 802 user navigation commands andtarget information, for example, from a remote control terminal such asdescribed herein. The navigation commands can be used for controllingnavigational parameters of the movable object such as the position,speed, orientation, or attitude of the movable object. In some cases,the navigation commands can include instructions for the movable objectto execute some pre-stored navigation commands such as those related toa predetermined navigation path. The navigation commands can be used tocontrol 804 the movable object to navigate according to a user-specifiedor pre-stored navigation path.

The target information can include initial target information (e.g.,position and/or size) of a particular target such as selected by a useroperating a control terminal and/or target type information such ascolor, texture, pattern, size, shape, dimension, and the like.

Target information can also include expected target information (e.g.,position and/or size). The expected target information specifies thecharacteristics that the target being tracked is expected to meet in theimages captured by the imaging device. The expected target informationmay be used to adjust the movable object, carrier and/or imaging deviceso that the target being tracked maintains parameters specified by theexpected target information. For example, the target may be tracked soas to maintain an expected position and/or size within one or moreimages captured by the imaging device. For example, the expectedposition of the tracked target may be near the center of the image oroff-center. The expected size of the tracked target may be around acertain number of pixels (e.g., the initial target size). The expectedtarget information may or may not be the same as the initial targetinformation. In various embodiments, expected target information may ormay not be provided by the control terminal. For example, expectedtarget information may be hardcoded in the control logic executed by aprocessing unit onboard the movable object, stored in a data store localand/or remote to the movable object, or obtained from other suitablesources.

In some embodiments, the target information (including specific targetinformation and target type information) may be generated at least inpart on user input at the control terminal 112. Additionally oralternatively, the target information may be generated based on datafrom other sources. For example, target type information may be derivedbased on previous images and/or extracted from local or remote datastores. Such target type information may be selected by the user and/orprovided automatically by default to the movable object.

The target information may be used to adjust 806 the movable object,carrier, and/or imaging device to track the target according to thetarget information. The tracking and any other related data processingmay be performed automatically by one or more processors onboard themovable object, one or more processors remote from the movable object,or any suitable combination thereof. In some embodiments, the targetinformation can be used to identify, by the movable object, the targetto be tracked. Such identification of the target may be performed basedon the initial target information including the specific characteristicsof a particular target (e.g., based on provided coordinates of thetarget within an image captured by the movable object), or generaltarget type information included in the target information.

Once a target is identified, expected target information (as part of thetarget information) can be used to detect a deviation from expectedcharacteristics of the target such as expected position and/or size. Thedeviation may be detected by comparing the target's characteristics(e.g., positions and/or sizes) within a series of images captured by themovable objects to determine whether the characteristics change acrossthe images. The series of images may be captured by the movable objectat different points in time. A change in position of the target may bedetected by comparing coordinates of the center of the target within animage to the coordinates of the expected target position. A change insize of the target may be detected by comparing the size of the area(e.g., in pixels) covered by the target with the expected target size.

Based at least in part on the detected deviation, control signals may begenerated automatically (e.g., by one or more processors onboard themovable object) that pertains to the movable object, the carrier, theimaging device, and/or any suitable component thereof. The controlsignals can be used to substantially correct the detected deviation. Insome embodiments, the adjustment may be used to substantially maintainone or more expected target characteristics (e.g., target positionand/or size) within the images captured by the movable object while themovable object navigates according to user provided navigation commands.The adjustment may be performed in substantially real time as themovable object is executing user-provided navigation commands orpre-stored navigation paths and/or as the imaging device is capturingone or more images. In some embodiments, the adjustment may be generatedbased on other information such as sensing data acquired by one or moresensors onboard the movable object (e.g., proximity sensor, or GPSsensor). For example, position information of the target being trackedmay be obtained by a proximity sensor and/or provided by the targetitself (e.g., GPS location). Such position information may be used, inaddition to the detected deviation, to generate the adjustment controlsignals to track the target.

In some embodiments, when the movable object is navigating accordinguser provided navigation commands or predetermined navigation paths, theadjustment for target tracking is limited to the carrier and/or theimaging device without affecting the state (e.g., attitude, position,orientation, or speed) of the movable object itself. This may be becauseit is infeasible or otherwise undesirable to adjust the movable objectwhile it executing user provided navigation commands or pre-storednavigation paths. For example, such adjustment may cause the carrier tomove the imaging device relative to the movable object such as around oralong one, two, or more rotational axes to compensate for deviation fromthe expected position of the target. Additionally or alternatively, theadjustment may cause adjustment to the focal length or zoom, focus, orother operational parameters of the imaging device to compensate fordeviation from the expected size of the target. Discussion of exemplarymethods for such adjustments is provided in connection with FIGS. 6-7.In other embodiments, the adjustment may also involve adjustment to themovable object.

According to another aspect of the present invention, methods andsystems are provided for controlling a movable object to navigateaccording to one or more navigation commands while tracking a target.FIG. 9 illustrates an exemplary process 900 for controlling a movableobject to navigate and track, in accordance with embodiments. Aspects ofthe process 900 may be performed by a control terminal such as describedherein. For example, the control terminal can be a handheld or wearabledevice. For example, the control terminal can include a smartphone,tablet, laptop, computer, glasses, gloves, helmet, microphone, orsuitable combinations thereof.

The process 900 includes receiving 902 user input related to navigationor navigation input. Such navigation input may be used to generatenavigation commands for controlling the movable object or a componentthereof such as described in connection with FIG. 8. The navigationinput may be provided via a user interface provided by the controlterminal, such as a keyboard, mouse, joystick, touchscreen, buttons,microphone, webcam, or the like. Any suitable user input can be used tointeract with the control device, such as manually entered commands,voice control, gesture control, or position control (e.g., via amovement, orientation and/or attitude of the control device).

The process 900 also includes receiving 904 user input related totracking or tracking input. Such tracking input may be used to generatetarget information related to a target or a type of targets to betracked by the movable object such as described herein. The trackinginput may be provided using the same or different interface as that forproviding the navigation input. For example, the tracking input may beprovided via a touchscreen while the navigation input may be providedvia a joystick or an inertial sensor configured to detect a change inattitude of the control terminal (e.g., rotating the control terminalaround a rotational axis). As another example, both the navigation inputand the tracking input may be provide via the same user interface suchvia a touchscreen, keyboard, mouse, or joystick.

In some embodiments, the navigation input, or the tracking input mayeach be provide via two or more user interfaces. For example, navigationinput may be provided via both joystick and touchscreen. As anotherexample, tracking input may be provided via touchscreen, joystick, andkeyboard.

The navigation input can be used to generate 906 navigation commands forthe movable object. For example, mechanical signals resulting from auser's movement of a joystick along or around one or more predefinedaxes may be translated into navigation commands for moving the movableobject along the corresponding axes (e.g., pitch, yaw, or roll). Asanother example, the user's interaction with a graphical user interface(e.g., movement of an icon representing an orientation or position ofthe movable object) may be converted to the corresponding navigationcommands. As yet another example, a user's movement of the controlterminal along or around one or more predefined axes may be detected byone or more inertial sensors and used to generate navigation commandsfor moving the movable object along the corresponding axes (e.g., pitch,yaw, or roll). In addition to navigation input, the navigation commandsmay be generated based on other data such as sensing data provided bythe movable object.

The tracking input can be used to generate 908 target information thatcan be used by the movable object to track one or more targets. Thetarget information may include initial target information and/orexpected target information such as described herein. Initial targetinformation can include specific characteristics of a particular target(e.g., based on provided coordinates of the target within an imagecaptured by the movable object), or general characteristics of a type oftargets (e.g., based on provided color and/or texture of the target(s)to be tracked). Initial target information may be used to identify thetarget to be tracked, for example, from one or more images. The expectedinformation can be used to detect a deviation from expectedcharacteristics of the target such as expected position and/or size. Insome embodiments, the expected information may be optional and some orall of the initial information may be used as the expected information.

The process 900 also includes receiving user input and generationcommands related to control of the carrier and/or imaging devices.Examples of such control may include changing an attitude and/ororientation of the imaging device via the carrier, adjusting one or moreoperational parameters (e.g., zoom) of the imaging device, and the like.

In some embodiments, such as described in further detail in FIG. 10, thecoordinates, size, and other characteristics of a target may be selectedby the user in a graphical user interface and used to generate initialand/or expected target information.

In some embodiments, the generated navigation commands and/or targetinformation may be provided 910 to the movable object, such as via acommunication system of a control terminal. Such commands andinformation may be provided while the movable object is in use (e.g.,moving or otherwise operating in the intended environment). In someembodiments, the navigation commands and target information may beprovided directly from a control terminal to a movable object. In someother embodiments, navigation commands and target information may beprovided to one or more intermediate entities (e.g., relay stations orother movable objects) before reaching the movable object.

In some embodiments, all of the steps of process 900 are implemented bythe same entity such as a single control terminal. In some other cases,the steps of process 900 are implemented by two or more separateentities. Such separate entities may communicate with each other, forexample, via wired or wireless connections. For example, the receiving902 of navigation input and generation 906 of navigation commands may beimplemented by first device while the receiving 904 of tracking inputand generation 908 of target information may be implemented by a seconddevice. The second device may transmit the target information to thefirst device, which may be configured to transmit both navigationcommands and target information to the movable object. As anotherexample, the receiving steps 902 and 904 may be performed by one device,while the generation steps 906 and 908 may be performed by anotherdevice.

According to some other aspects of the present invention, an interactiveuser interface is provided for selecting and viewing, in substantiallyreal time, targets tracked by a movable object described herein such asa UAV. Such a user interface may be provided by a control terminal suchas described herein. FIG. 10 illustrates an exemplary process 1000 forselecting a target, in accordance with embodiments. Aspects of theprocess 1000 may be performed by a control terminal such as describedherein.

The process 1000 includes displaying 1002 one or more images captured bythe movable objects. The images may be captured by an imaging devicecoupled to the movable object via a carrier as described herein. Thecarrier may or may not permit the imaging device to move relative to themovable object. The imaging device may alternatively be coupled directlyto the movable object without a carrier. The images may be transmittedto a control terminal from the movable object or directly from theimaging device. The images thus received by the control terminal may bedecompressed or otherwise processed before being displayed on a userinterface associated with the control terminal. The user interface maybe provided by a display that is integral to the control terminal.Alternatively, the display may be separate from but operativelyconnected to the control terminal. The images thus displayed may bestatic (e.g., still images) or dynamic (e.g., videos).

In some embodiments, the images may be displayed or streamed insubstantially real time as the images are received from the movableobject and/or the imaging device. In some embodiments, the images aretransmitted from the movable object and/or imaging device insubstantially real time as the images are captured by the imagingdevice. In other cases, there may be some delay between the captureand/or transmission of the images and the display of the images.

In some embodiments, the process 1000 includes receiving 1004 a userselection of a target from within at least one of the images beingdisplayed. The user may select the target via the same user interfacethat displays the images. For example, in some embodiments, the userselection of the target is achieved by a user selecting an area of theat least one of the one or more images being displayed on the display,the selected area corresponding to the target. For example, the user mayselect the target by directly touching a touchscreen using a finger orstylus. As another example, the user may select the target using amouse, joystick, gesture, or voice command. In yet another embodiment,the user may select target via a wearable device such as a helmet,virtual reality goggle, and the like. Selection of the target area caninvolve any suitable motion including touching or tapping, swiping,circling, clicking, or the like associated with any suitable inputmethods such as described herein.

In some embodiments, the selected target is displayed with a selectionindicator in the images to indicate that is selected by the user. Theselection indicator may be represented by a circle, checkmark, arrow,polygon, or any other suitably shaped object graphical object of anysuitable color or pattern.

The user-selected target can be used to generate 1006 targetinformation. For example, the coordinates and pixel size of the selectedtarget within an image can be used to as or used to generate initialinformation for the target. Alternatively or additionally,characteristics associated with the selected target can be used togenerate at least some of the target type information. In someembodiments, the remaining target type information may be provided basedon other user input. For example, the target type information mayinclude the color and texture of the target to be tracked. The color ofthe target type information may be extracted from a particular targetselected by the user; whereas the texture of the target may be manuallyentered by the user, or selected from a plurality of predeterminedpatterns.

In some embodiments, a user is allowed to select between a manualtracking mode and an automatic tracking mode. When the manual trackingmode is selected, a user can specify a specific target to track such asfrom an image being displayed by the control terminal. On the otherhand, when the automatic tracking mode is selected, the user can specifythe target type information rather than a specific target to be tracked.The movable object can then use the target type information toautomatically identify the target to be tracked and subsequently trackthe identified target.

Optionally, information about the selected target may be used togenerate expected target information. For example, the initial targetinformation may be the same the expected target information. As anotherexample, the expected target information may be expressed with referenceto the initial target information. For example, the expected targetposition may be expressed using an offset from the initial targetposition. Similarly, the expected target size may be expressed as aratio between the expected target size and the initial target size(e.g., the expected target size may be twice the size of the initialtarget size). In various embodiments, the target information may begenerated based on a target selected by a user, data obtained usingother user input methods (e.g., data entry from keyboard) and/or datafrom non-user sources (e.g., local or remote data store or othersuitable computing systems).

The target information can be provided 1008 to the movable object fortracking purposes. For example, initial target information and/or targettype information can be used by one or more processors on the movableobject to identify the target to be tracked. The expected targetinformation can be used by the movable object to substantially maintainpredetermined position, size, and other characteristics of the targetduring the tracking process.

In various embodiments, any portion of the target information used tocontrol the tracking process can be provided by users and the rest bythe tracking system. For example, in an embodiment, initial targetinformation is provided by a user via user selection of a concretetarget whereas the expected target information is provided by the systemby default. In another embodiment, a user may provide the target typeinformation whereas the system automatically determines the target basedon the target type information and tracks the target usinguser-specified or system-provided expected target information. In yetanother embodiment, a user may supply all the target informationincluding the initial and expected target information and the systemsimply tracks the specified target accordingly.

The specific allocation the tracking control between the users and thetracking system can vary depending on a variety of factors such as thesurroundings of the movable object, speed or altitude of the movableobject, user preferences, and computing resources (e.g., CPU or memory)available onboard and/or off-board the movable object, the like. Forexample, relatively more control may be allocated to the user when themovable object is navigating in a relatively complex environment (e.g.,with numerous buildings or obstacles or indoor) than when the movableobject is navigating in a relatively simple environment (e.g., wide openspace or outdoor). As another example, more control may be allocated tothe user when the movable object is at a lower altitude than when themovable object is at a higher altitude. As yet another example, morecontrol may be allocated to the movable object if the movable object isequipped with a high-speed processor adapted to perform complexcomputations relatively quickly.

FIG. 11 illustrates an exemplary process 1100 for viewing a trackedtarget, in accordance with embodiments. Aspects of the process 1100 maybe performed by the same or a different control terminal forimplementing the process 1000.

The process 1100 includes receiving 1102 images captured by a movableobject such as described herein (e.g., a UAV). The images may becaptured by an imaging device coupled to the movable object (e.g., via acarrier). In some cases, the images may be received directly from theimaging device that captured the images. The images may be received by acontrol terminal such as described herein. In some embodiments, imagesmay be received after target information such as described herein hasbeen provided to the movable object. The target information may havebeen provided according to a process similar to the process 900 or 1000such as discussed in FIG. 9 or 10. The movable object may be configuredto track the target as identified and/or specified by the targetinformation according to a process similar to the process 400 discussedin FIG. 4. As such, at least some of the images received may include thetarget being tracked.

The process 1100 also includes receiving 1104 tracking data orinformation used to indicate characteristics of one or more targettracked by the movable object. Examples of tracking information mayinclude relative or absolute coordinates and/or size of the targetwithin one or more images received from the movable object, changes oroffsets of the target between consecutive image frames, GPS coordinates,or other positional information of the target, and the like.

The received images may be displayed 1106 with tracking data to show thetarget that is being tracked by the movable object. For example, thetracking data may be used to determine the location and/or size of thetarget being tracked in some or all of the images. Based on thisinformation, a tracking indicator such as similar to the selectionindicator discussed in FIG. 10 may be generated and displayed. Forexample, given the target location within a given image, the image maybe displayed with a tracking indicator at approximately the targetlocation and of a size large enough to cover the target or a portionthereof. Examples of a tracking indicator may include a box, a circle,an oval, or a graphical object of any other suitable shape, color, orsize.

Advantageously, displaying tracked target with such tracking indicatorhelps the users to visually identify the target being tracked whileviewing the images that are captured by and/or transmitted from themovable object in substantially real time. In other words, the users mayview the targets substantially concurrently as they are being tracked bythe movable object. In other embodiments, the display may be providedafter some delay.

In some embodiments, while the viewing the target being tracked by themovable object, the user may select the target being tracked (e.g., asindicated by a tracking indicator) to view more detailed informationwith respect to the target. The user may also perform other operationswith respect to the target or any other objects being displayed byinteracting with the user interface where the images are displayed. Forexample, the user may zoom in or out by moving fingers across atouchscreen.

Some of such user operations may be converted to tracking commands andsent to the movable object. For example, user's zoom in/out operationsas described above may be translated to suitable zoom in/out commandsfor the imaging device and/or control commands to move the movableobject closer to or farther away from the target. Alternatively oradditionally, the user may elect to switch to a different target fortracking. Such a switch may be performed dynamically as the user selectsthe different target. The target information related to the differenttarget may be dynamically provided to the movable object, for example,in a manner similar to that described in FIG. 10. In response, themovable object may then dynamically switch to tracking the differenttarget using the methods described herein, for example, in substantiallyreal-time.

FIG. 12 illustrates an exemplary control terminal 1200 for controlling amovable object, in accordance with embodiments. The control terminal1200 includes a display area 1202 for displaying sensing data such asimage data captured by an imaging device or image sensor coupled to amovable object such as a UAV. The image data may include images of oneor more objects 1210 and 1206 as captured by an imaging device or imagesensor onboard the movable object. One of the one or more objects caninclude a target 1206 that is being tracked by the movable object. Thetarget may be displayed with an indicator 1208. The indicator 1208 mayinclude a selection indicator such as discussed in FIG. 10 to indicatethat the target is selected by the user to be tracked by the movableobject, and/or a tracking indicator such as discussed in FIG. 11 toindicate that the target is being tracked by the movable object. Theindicator 1208 may be configured to move with the target as itspositions change across different images.

The control terminal 1200 can also include one or more input devices1204 and 1205 for receiving user input. Such input device may includejoysticks 1204, buttons or switches 1205, keyboard, mouse, touchscreenor any suitable mechanisms. For example, one or more buttons or switches1205 may be used to switch between different input modes such asnavigation input mode, target input mode, and the like. The switches1205 may also be used to select between a manual tracking mode and anautomatic tracking mode as described herein. The functionalitiesassociated with some of the input devices such as the joysticks 1204 maychange depending on the selected input mode. For example, such inputdevices may be used to provide navigation input when the navigationinput mode is selected and target information when the target input modeis selected. As another example, the input devices may be used to selectspecific target when the manual tracking mode is selected and to providetarget type information when the auto tracking mode is selected.

FIGS. 13A-C illustrate exemplary methods for selecting a target using auser interface, in accordance with some embodiments. In an embodiment,such as illustrated in FIG. 13A, the user interface can enable a user toselect a target 1304 by touching, using a hand 1302, an area of atouchscreen that corresponds to the target. The selected target 1304 maybe indicated by a selection indicator 1306, such as a circle surroundingthe selected target 1304. In another embodiment, such as illustrated inFIG. 13B, the user interface can enable a user to select a target 1310using a mouse 1308, for example, by clicking on an area of a screen thatcorresponds to the target. The selected target 1310 may be indicatedwith a selection indicator 1312. In yet another embodiment, such asillustrated in FIG. 13C, the user interface can enable a user to selecta target 1316 using a stylus 1314, for example, by tapping on an area ofa screen that corresponds to the target. The selected target 1316 may beindicated with a selection indicator 1318. In various embodiments, theselection indicator may be of any arbitrary shape such as a circle 1306,rectangle 1312, or diamond 1318.

The systems and methods described herein can be implemented by and/orapplied to a wide variety of movable objects. The systems, devices, andmethods described herein can be applied to a wide variety of movableobjects. As previously mentioned, any description herein of an aerialvehicle may apply to and be used for any movable object. A movableobject of the present invention can be configured to move within anysuitable environment, such as in air (e.g., a fixed-wing aircraft, arotary-wing aircraft, or an aircraft having neither fixed wings norrotary wings), in water (e.g., a ship or a submarine), on ground (e.g.,a motor vehicle, such as a car, truck, bus, van, motorcycle; a movablestructure or frame such as a stick, fishing pole; or a train), under theground (e.g., a subway), in space (e.g., a spaceplane, a satellite, or aprobe), or any combination of these environments. The movable object canbe a vehicle, such as a vehicle described elsewhere herein. In someembodiments, the movable object can be mounted on a living subject, suchas a human or an animal. Suitable animals can include primates, avines,canines, felines, equines, bovines, ovines, porcines, delphines,rodents, or insects.

The movable object may be capable of moving freely within theenvironment with respect to six degrees of freedom (e.g., three degreesof freedom in translation and three degrees of freedom in rotation).Alternatively, the movement of the movable object can be constrainedwith respect to one or more degrees of freedom, such as by apredetermined path, track, or orientation. The movement can be actuatedby any suitable actuation mechanism, such as an engine or a motor. Theactuation mechanism of the movable object can be powered by any suitableenergy source, such as electrical energy, magnetic energy, solar energy,wind energy, gravitational energy, chemical energy, nuclear energy, orany suitable combination thereof. The movable object may beself-propelled via a propulsion system, as described elsewhere herein.The propulsion system may optionally run on an energy source, such aselectrical energy, magnetic energy, solar energy, wind energy,gravitational energy, chemical energy, nuclear energy, or any suitablecombination thereof. Alternatively, the movable object may be carried bya living being.

In some instances, the movable object can be a vehicle. Suitablevehicles may include water vehicles, aerial vehicles, space vehicles, orground vehicles. For example, aerial vehicles may be fixed-wing aircraft(e.g., airplane, gliders), rotary-wing aircraft (e.g., helicopters,rotorcraft), aircraft having both fixed wings and rotary wings, oraircraft having neither (e.g., blimps, hot air balloons). A vehicle canbe self-propelled, such as self-propelled through the air, on or inwater, in space, or on or under the ground. A self-propelled vehicle canutilize a propulsion system, such as a propulsion system including oneor more engines, motors, wheels, axles, magnets, rotors, propellers,blades, nozzles, or any suitable combination thereof. In some instances,the propulsion system can be used to enable the movable object to takeoff from a surface, land on a surface, maintain its current positionand/or orientation (e.g., hover), change orientation, and/or changeposition.

The movable object can be controlled remotely by a user or controlledlocally by an occupant within or on the movable object. In someembodiments, the movable object is an unmanned movable object, such as aUAV. An unmanned movable object, such as a UAV, may not have an occupantonboard the movable object. The movable object can be controlled by ahuman or an autonomous control system (e.g., a computer control system),or any suitable combination thereof. The movable object can be anautonomous or semi-autonomous robot, such as a robot configured with anartificial intelligence.

The movable object can have any suitable size and/or dimensions. In someembodiments, the movable object may be of a size and/or dimensions tohave a human occupant within or on the vehicle. Alternatively, themovable object may be of size and/or dimensions smaller than thatcapable of having a human occupant within or on the vehicle. The movableobject may be of a size and/or dimensions suitable for being lifted orcarried by a human. Alternatively, the movable object may be larger thana size and/or dimensions suitable for being lifted or carried by ahuman. In some instances, the movable object may have a maximumdimension (e.g., length, width, height, diameter, diagonal) of less thanor equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Themaximum dimension may be greater than or equal to about: 2 cm, 5 cm, 10cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance betweenshafts of opposite rotors of the movable object may be less than orequal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.Alternatively, the distance between shafts of opposite rotors may begreater than or equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m,or 10 m.

In some embodiments, the movable object may have a volume of less than100 cm×100 cm×100 cm, less than 50 cm×50 cm×30 cm, or less than 5 cm×5cm×3 cm. The total volume of the movable object may be less than orequal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40 cm³, 50cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³, 300 cm,500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³, 1 m³, or10 m³. Conversely, the total volume of the movable object may be greaterthan or equal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40cm³, 50 cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³,300 cm, 500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³, 1m³, or 10 m³.

In some embodiments, the movable object may have a footprint (which mayrefer to the lateral cross-sectional area encompassed by the movableobject) less than or equal to about: 32,000 cm², 20,000 cm², 10,000 cm²,1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm². Conversely, thefootprint may be greater than or equal to about: 32,000 cm², 20,000 cm²,10,000 cm², 1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm².

In some instances, the movable object may weigh no more than 1000 kg.The weight of the movable object may be less than or equal to about:1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg,8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg,or 0.01 kg. Conversely, the weight may be greater than or equal toabout: 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1kg, 0.05 kg, or 0.01 kg.

In some embodiments, a movable object may be small relative to a loadcarried by the movable object. The load may include a payload and/or acarrier, as described in further detail below. In some examples, a ratioof a movable object weight to a load weight may be greater than, lessthan, or equal to about 1:1. In some instances, a ratio of a movableobject weight to a load weight may be greater than, less than, or equalto about 1:1. Optionally, a ratio of a carrier weight to a load weightmay be greater than, less than, or equal to about 1:1. When desired, theratio of an movable object weight to a load weight may be less than orequal to: 1:2, 1:3, 1:4, 1:5, 1:10, or even less. Conversely, the ratioof a movable object weight to a load weight can also be greater than orequal to: 2:1, 3:1, 4:1, 5:1, 10:1, or even greater.

In some embodiments, the movable object may have low energy consumption.For example, the movable object may use less than about: 5 W/h, 4 W/h, 3W/h, 2 W/h, 1 W/h, or less. In some instances, a carrier of the movableobject may have low energy consumption. For example, the carrier may useless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less. Optionally,a payload of the movable object may have low energy consumption, such asless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less.

FIG. 14 illustrates a UAV 1400, in accordance with embodiments of thepresent invention. The UAV may be an example of a movable object asdescribed herein. The UAV 1400 can include a propulsion system havingfour rotors 1402, 1404, 1406, and 1408. Any number of rotors may beprovided (e.g., one, two, three, four, five, six, or more). The rotorscan be embodiments of the self-tightening rotors described elsewhereherein. The rotors, rotor assemblies, or other propulsion systems of theunmanned aerial vehicle may enable the unmanned aerial vehicle tohover/maintain position, change orientation, and/or change location. Thedistance between shafts of opposite rotors can be any suitable length1410. For example, the length 1410 can be less than or equal to 2 m, orless than equal to 5 m. In some embodiments, the length 1410 can bewithin a range from 40 cm to 1 m, from 10 cm to 2 m, or from 5 cm to 5m. Any description herein of a UAV may apply to a movable object, suchas a movable object of a different type, and vice versa.

In some embodiments, the movable object can be configured to carry aload. The load can include one or more of passengers, cargo, equipment,instruments, and the like. The load can be provided within a housing.The housing may be separate from a housing of the movable object, or bepart of a housing for an movable object. Alternatively, the load can beprovided with a housing while the movable object does not have ahousing. Alternatively, portions of the load or the entire load can beprovided without a housing. The load can be rigidly fixed relative tothe movable object. Optionally, the load can be movable relative to themovable object (e.g., translatable or rotatable relative to the movableobject).

In some embodiments, the load includes a payload 1412. The payload canbe configured not to perform any operation or function. Alternatively,the payload can be a payload configured to perform an operation orfunction, also known as a functional payload. For example, the payloadcan include one or more sensors for surveying one or more targets. Anysuitable sensor can be incorporated into the payload, such as an imagecapture device (e.g., a camera), an audio capture device (e.g., aparabolic microphone), an infrared imaging device, or an ultravioletimaging device. The sensor can provide static sensing data (e.g., aphotograph) or dynamic sensing data (e.g., a video). In someembodiments, the sensor provides sensing data for the target of thepayload. Alternatively or in combination, the payload can include one ormore emitters for providing signals to one or more targets. Any suitableemitter can be used, such as an illumination source or a sound source.In some embodiments, the payload includes one or more transceivers, suchas for communication with a module remote from the movable object.Optionally, the payload can be configured to interact with theenvironment or a target. For example, the payload can include a tool,instrument, or mechanism capable of manipulating objects, such as arobotic arm.

Optionally, the load may include a carrier 1414. The carrier can beprovided for the payload and the payload can be coupled to the movableobject via the carrier, either directly (e.g., directly contacting themovable object) or indirectly (e.g., not contacting the movable object).Conversely, the payload can be mounted on the movable object withoutrequiring a carrier. The payload can be integrally formed with thecarrier. Alternatively, the payload can be releasably coupled to thecarrier. In some embodiments, the payload can include one or morepayload elements, and one or more of the payload elements can be movablerelative to the movable object and/or the carrier, as described above.

The carrier can be integrally formed with the movable object.Alternatively, the carrier can be releasably coupled to the movableobject. The carrier can be coupled to the movable object directly orindirectly. The carrier can provide support to the payload (e.g., carryat least part of the weight of the payload). The carrier can include asuitable mounting structure (e.g., a gimbal platform) capable ofstabilizing and/or directing the movement of the payload. In someembodiments, the carrier can be adapted to control the state of thepayload (e.g., position and/or orientation) relative to the movableobject. For example, the carrier can be configured to move relative tothe movable object (e.g., with respect to one, two, or three degrees oftranslation and/or one, two, or three degrees of rotation) such that thepayload maintains its position and/or orientation relative to a suitablereference frame regardless of the movement of the movable object. Thereference frame can be a fixed reference frame (e.g., the surroundingenvironment). Alternatively, the reference frame can be a movingreference frame (e.g., the movable object, a payload target).

In some embodiments, the carrier can be configured to permit movement ofthe payload relative to the carrier and/or movable object. The movementcan be a translation with respect to up to three degrees of freedom(e.g., along one, two, or three axes) or a rotation with respect to upto three degrees of freedom (e.g., about one, two, or three axes), orany suitable combination thereof.

In some instances, the carrier can include a carrier frame assembly anda carrier actuation assembly. The carrier frame assembly can providestructural support to the payload. The carrier frame assembly caninclude individual carrier frame components, some of which can bemovable relative to one another. The carrier actuation assembly caninclude one or more actuators (e.g., motors) that actuate movement ofthe individual carrier frame components. The actuators can permit themovement of multiple carrier frame components simultaneously, or may beconfigured to permit the movement of a single carrier frame component ata time. The movement of the carrier frame components can produce acorresponding movement of the payload. For example, the carrieractuation assembly can actuate a rotation of one or more carrier framecomponents about one or more axes of rotation (e.g., roll axis, pitchaxis, or yaw axis). The rotation of the one or more carrier framecomponents can cause a payload to rotate about one or more axes ofrotation relative to the movable object. Alternatively or incombination, the carrier actuation assembly can actuate a translation ofone or more carrier frame components along one or more axes oftranslation, and thereby produce a translation of the payload along oneor more corresponding axes relative to the movable object.

In some embodiments, the movement of the movable object, carrier, andpayload relative to a fixed reference frame (e.g., the surroundingenvironment) and/or to each other, can be controlled by a terminal. Theterminal can be a remote control device at a location distant from themovable object, carrier, and/or payload. The terminal can be disposed onor affixed to a support platform. Alternatively, the terminal can be ahandheld or wearable device. For example, the terminal can include asmartphone, tablet, laptop, computer, glasses, gloves, helmet,microphone, or suitable combinations thereof. The terminal can include auser interface, such as a keyboard, mouse, joystick, touchscreen, ordisplay. Any suitable user input can be used to interact with theterminal, such as manually entered commands, voice control, gesturecontrol, or position control (e.g., via a movement, location or tilt ofthe terminal).

The terminal can be used to control any suitable state of the movableobject, carrier, and/or payload. For example, the terminal can be usedto control the position and/or orientation of the movable object,carrier, and/or payload relative to a fixed reference from and/or toeach other. In some embodiments, the terminal can be used to controlindividual elements of the movable object, carrier, and/or payload, suchas the actuation assembly of the carrier, a sensor of the payload, or anemitter of the payload. The terminal can include a wirelesscommunication device adapted to communicate with one or more of themovable object, carrier, or payload.

The terminal can include a suitable display unit for viewing informationof the movable object, carrier, and/or payload. For example, theterminal can be configured to display information of the movable object,carrier, and/or payload with respect to position, translationalvelocity, translational acceleration, orientation, angular velocity,angular acceleration, or any suitable combinations thereof. In someembodiments, the terminal can display information provided by thepayload, such as data provided by a functional payload (e.g., imagesrecorded by a camera or other image capturing device).

Optionally, the same terminal may both control the movable object,carrier, and/or payload, or a state of the movable object, carrierand/or payload, as well as receive and/or display information from themovable object, carrier and/or payload. For example, a terminal maycontrol the position of the payload relative to an environment, whiledisplaying image data captured by the payload, or information about theposition of the payload. Alternatively, different terminals may be usedfor different functions. For example, a first terminal may controlmovement or a state of the movable object, carrier, and/or payload whilea second terminal may receive and/or display information from themovable object, carrier, and/or payload. For example, a first terminalmay be used to control the positioning of the payload relative to anenvironment while a second terminal displays image data captured by thepayload. Various communication modes may be utilized between a movableobject and an integrated terminal that both controls the movable objectand receives data, or between the movable object and multiple terminalsthat both control the movable object and receives data. For example, atleast two different communication modes may be formed between themovable object and the terminal that both controls the movable objectand receives data from the movable object.

FIG. 15 illustrates a movable object 1500 including a carrier 1502 and apayload 1504, in accordance with embodiments. Although the movableobject 1500 is depicted as an aircraft, this depiction is not intendedto be limiting, and any suitable type of movable object can be used, aspreviously described herein. One of skill in the art would appreciatethat any of the embodiments described herein in the context of aircraftsystems can be applied to any suitable movable object (e.g., a UAV). Insome instances, the payload 1504 may be provided on the movable object1500 without requiring the carrier 1502. The movable object 1500 mayinclude propulsion mechanisms 1506, a sensing system 1508, and acommunication system 1510.

The propulsion mechanisms 1506 can include one or more of rotors,propellers, blades, engines, motors, wheels, axles, magnets, or nozzles,as previously described. For example, the propulsion mechanisms 1506 maybe self-tightening rotors, rotor assemblies, or other rotary propulsionunits, as disclosed elsewhere herein. The movable object may have one ormore, two or more, three or more, or four or more propulsion mechanisms.The propulsion mechanisms may all be of the same type. Alternatively,one or more propulsion mechanisms can be different types of propulsionmechanisms. The propulsion mechanisms 1506 can be mounted on the movableobject 1500 using any suitable means, such as a support element (e.g., adrive shaft) as described elsewhere herein. The propulsion mechanisms1506 can be mounted on any suitable portion of the movable object 1500,such on the top, bottom, front, back, sides, or suitable combinationsthereof.

In some embodiments, the propulsion mechanisms 1506 can enable themovable object 1500 to take off vertically from a surface or landvertically on a surface without requiring any horizontal movement of themovable object 1500 (e.g., without traveling down a runway). Optionally,the propulsion mechanisms 1506 can be operable to permit the movableobject 1500 to hover in the air at a specified position and/ororientation. One or more of the propulsion mechanism 1500 may becontrolled independently of the other propulsion mechanisms.Alternatively, the propulsion mechanisms 1500 can be configured to becontrolled simultaneously. For example, the movable object 1500 can havemultiple horizontally oriented rotors that can provide lift and/orthrust to the movable object. The multiple horizontally oriented rotorscan be actuated to provide vertical takeoff, vertical landing, andhovering capabilities to the movable object 1500. In some embodiments,one or more of the horizontally oriented rotors may spin in a clockwisedirection, while one or more of the horizontally rotors may spin in acounterclockwise direction. For example, the number of clockwise rotorsmay be equal to the number of counterclockwise rotors. The rotation rateof each of the horizontally oriented rotors can be varied independentlyin order to control the lift and/or thrust produced by each rotor, andthereby adjust the spatial disposition, velocity, and/or acceleration ofthe movable object 1500 (e.g., with respect to up to three degrees oftranslation and up to three degrees of rotation).

The sensing system 1508 can include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 1500 (e.g., with respect to up to three degrees of translationand up to three degrees of rotation). The one or more sensors caninclude any of the sensors previously described herein, including GPSsensors, motion sensors, inertial sensors, proximity sensors, or imagesensors. The sensing data provided by the sensing system 1508 can beused to control the spatial disposition, velocity, and/or orientation ofthe movable object 1500 (e.g., using a suitable processing unit and/orcontrol module, as described below). Alternatively, the sensing system1508 can be used to provide data regarding the environment surroundingthe movable object, such as weather conditions, proximity to potentialobstacles, location of geographical features, location of manmadestructures, and the like.

The communication system 1510 enables communication with terminal 1512having a communication system 1514 via wireless signals 1516. Thecommunication systems 1510, 1514 may include any number of transmitters,receivers, and/or transceivers suitable for wireless communication. Thecommunication may be one-way communication, such that data can betransmitted in only one direction. For example, one-way communicationmay involve only the movable object 1500 transmitting data to theterminal 1512, or vice-versa. The data may be transmitted from one ormore transmitters of the communication system 1510 to one or morereceivers of the communication system 1512, or vice-versa.Alternatively, the communication may be two-way communication, such thatdata can be transmitted in both directions between the movable object1500 and the terminal 1512. The two-way communication can involvetransmitting data from one or more transmitters of the communicationsystem 1510 to one or more receivers of the communication system 1514,and vice-versa.

In some embodiments, the terminal 1512 can provide control data to oneor more of the movable object 1500, carrier 1502, and payload 1504 andreceive information from one or more of the movable object 1500, carrier1502, and payload 1504 (e.g., position and/or motion information of themovable object, carrier or payload; data sensed by the payload such asimage data captured by a payload camera). In some instances, controldata from the terminal may include instructions for relative positions,movements, actuations, or controls of the movable object, carrier,and/or payload. For example, the control data may result in amodification of the location and/or orientation of the movable object(e.g., via control of the propulsion mechanisms 1506), or a movement ofthe payload with respect to the movable object (e.g., via control of thecarrier 1502). The control data from the terminal may result in controlof the payload, such as control of the operation of a camera or otherimage capturing device (e.g., taking still or moving pictures, zoomingin or out, turning on or off, switching imaging modes, change imageresolution, changing focus, changing depth of field, changing exposuretime, changing viewing angle or field of view). In some instances, thecommunications from the movable object, carrier and/or payload mayinclude information from one or more sensors (e.g., of the sensingsystem 1508 or of the payload 1504). The communications may includesensed information from one or more different types of sensors (e.g.,GPS sensors, motion sensors, inertial sensor, proximity sensors, orimage sensors). Such information may pertain to the position (e.g.,location, orientation), movement, or acceleration of the movable object,carrier, and/or payload. Such information from a payload may includedata captured by the payload or a sensed state of the payload. Thecontrol data provided transmitted by the terminal 1512 can be configuredto control a state of one or more of the movable object 1500, carrier1502, or payload 1504. Alternatively or in combination, the carrier 1502and payload 1504 can also each include a communication module configuredto communicate with terminal 1512, such that the terminal cancommunicate with and control each of the movable object 1500, carrier1502, and payload 1504 independently.

In some embodiments, the movable object 1500 can be configured tocommunicate with another remote device in addition to the terminal 1512,or instead of the terminal 1512. The terminal 1512 may also beconfigured to communicate with another remote device as well as themovable object 1500. For example, the movable object 1500 and/orterminal 1512 may communicate with another movable object, or a carrieror payload of another movable object. When desired, the remote devicemay be a second terminal or other computing device (e.g., computer,laptop, tablet, smartphone, or other mobile device). The remote devicecan be configured to transmit data to the movable object 1500, receivedata from the movable object 1500, transmit data to the terminal 1512,and/or receive data from the terminal 1512. Optionally, the remotedevice can be connected to the Internet or other telecommunicationsnetwork, such that data received from the movable object 1500 and/orterminal 1512 can be uploaded to a website or server.

FIG. 16 illustrates an exemplary system 1600 for tracking a target, inaccordance with embodiments. The system 1600 can be used in combinationwith any suitable embodiment of the systems, devices, and methodsdisclosed herein. For example, the system 1600 may implemented orcarried by a movable object. The system 1600 can include a sensingmodule 1602, processing unit 1604, non-transitory computer readablemedium 1606, control module 1608, and communication module 1610.

The sensing module 1602 can utilize different types of sensors thatcollect information relating to the movable objects in different ways.Different types of sensors may sense different types of signals orsignals from different sources. For example, the sensors can includeinertial sensors, GPS sensors, proximity sensors (e.g., lidar), orvision/image sensors (e.g., a camera). The sensing module 1602 can beoperatively coupled to a processing unit 1604 having a plurality ofprocessors. In some embodiments, the sensing module can be operativelycoupled to a transmission module 1612 (e.g., a Wi-Fi image transmissionmodule) configured to directly transmit sensing data to a suitableexternal device or system. For example, the transmission module 1612 canbe used to transmit images captured by a camera of the sensing module1602 to a remote terminal.

The processing unit 1604 can have one or more processors, such as aprogrammable processor (e.g., a central processing unit (CPU)). Forexample, the processing unit 1604 may include a field programmable gatearray (FPGA) and/or one or more ARM processors. The processing unit 1604can be operatively coupled to a non-transitory computer readable medium1606. The non-transitory computer readable medium 1606 can store logic,code, and/or program instructions executable by the processing unit 1604for performing one or more steps. The non-transitory computer readablemedium can include one or more memory units (e.g., removable media orexternal storage such as an SD card or random access memory (RAM)). Insome embodiments, data from the sensing module 1602 can be directlyconveyed to and stored within the memory units of the non-transitorycomputer readable medium 1606. The memory units of the non-transitorycomputer readable medium 1606 can store logic, code and/or programinstructions executable by the processing unit 1604 to perform anysuitable embodiment of the methods described herein. For example, theprocessing unit 1604 can be configured to execute instructions causingone or more processors of the processing unit 1604 to perform thetracking functionalities discussed herein. The memory units can storesensing data from the sensing module to be processed by the processingunit 1604. In some embodiments, the memory units of the non-transitorycomputer readable medium 1606 can be used to store the processingresults produced by the processing unit 1604.

In some embodiments, the processing unit 1604 can be operatively coupledto a control module 1608 configured to control a state of the movableobject. For example, the control module 1608 can be configured tocontrol the propulsion mechanisms of the movable object to adjust thespatial disposition, velocity, and/or acceleration of the movable objectwith respect to six degrees of freedom. Alternatively or in combination,the control module 1608 can control one or more of a state of a carrier,payload, or sensing module.

The processing unit 1604 can be operatively coupled to a communicationmodule 1610 configured to transmit and/or receive data from one or moreexternal devices (e.g., a terminal, display device, or other remotecontroller). Any suitable means of communication can be used, such aswired communication or wireless communication. For example, thecommunication module 1610 can utilize one or more of local area networks(LAN), wide area networks (WAN), infrared, radio, WiFi, point-to-point(P2P) networks, telecommunication networks, cloud communication, and thelike. Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. The communication module1610 can transmit and/or receive one or more of sensing data from thesensing module 1602, and/or processing results produced by theprocessing unit 1604, predetermined control data or user commands from aterminal or remote controller, and the like.

The components of the system 1600 can be arranged in any suitableconfiguration. For example, one or more of the components of the system1600 can be located on the movable object, carrier, payload, terminal,sensing system, or an additional external device in communication withone or more of the above. Additionally, although FIG. 16 depicts asingle processing unit 1604 and a single non-transitory computerreadable medium 1606, one of skill in the art would appreciate that thisis not intended to be limiting, and that the system 1600 can include aplurality of processing units and/or non-transitory computer readablemedia. In some embodiments, one or more of the plurality of processingunits and/or non-transitory computer readable media can be situated atdifferent locations, such as on the movable object, carrier, payload,terminal, sensing module, additional external device in communicationwith one or more of the above, or suitable combinations thereof, suchthat any suitable aspect of the processing and/or memory functionsperformed by the system 1600 can occur at one or more of theaforementioned locations.

FIG. 17 illustrates an exemplary system 1700 for controlling a movableobject, in accordance with embodiments. The system 1700 can be used tocontrol a movable object such as a UAV. The system 1700 can be used incombination with any suitable embodiment of the systems, devices, andmethods disclosed herein. For example, the system 1700 may beimplemented by a control terminal such as described herein. The system1700 can include an input module 1702, a processing unit 1704, anon-transitory computer readable medium 1706, a display module 1708, anda communication module 1710, all interconnected via a bus or a similarnetwork.

The input module 1702 can include one or more input mechanisms forreceiving input from a user operating the input module. Such inputmechanisms can include one or more joysticks, switches, knobs, slideswitches, buttons, dials, touchscreens, keypads, keyboard, mouse, voicecontrols, gesture controls, inertial sensors, and the like. Such inputmodule 1702 can be used to receive user input used to control aspects ofthe movable object, carrier, payload, or a component thereof. Suchaspects may include attitude, position, orientation, navigation,tracking, and the like. For example, the input mechanisms can bemanually set by a user to one or more positions, each of the positionscorresponding to a predetermined input for controlling the UAV.

In some embodiments, the input mechanisms can be manipulated by a userto input control commands for controlling the navigation of the movableobject. For instance, the user may utilize a knob, switch, or a similarinput mechanism to input a flight mode for the movable object such asauto pilot or navigation according to a predetermined navigation path.As another example, the user may control the position, attitude,orientation and other aspect of the movable object by the tilting thecontrol terminal in certain ways, which may be detected by one or moreinertial sensors and used to generate the corresponding navigationcommands. As yet another example, the user may use the input mechanismto adjust an operational parameter (e.g., zoom) of the payload, anattitude and/or the payload (e.g., via the carrier), or other aspects ofany object onboard the movable object.

In some embodiments, the input mechanisms can be manipulated by a userto input target information such as described herein. For instance, theuser may utilize a knob, switch, or a similar input mechanism to selectthe appropriate tracking mode such as manual tracking mode or autotracking mode. The user may also use the input mechanisms to select aparticular target to track, specify target type information, and othersimilar information. In various embodiments, the input module may beimplemented by more than one device. For example, the input module maybe implemented by a standard remote controller with joystick(s) that isoperatively coupled with a mobile device (e.g., smartphone) runningsuitable mobile application software (“app”) that can generate controlinstructions for the movable object. The app may be configured to obtaininputs from a user.

The processing unit 1704 can have one or more processors, such as aprogrammable processor (e.g., a central processing unit (CPU) or amicrocontroller). The processing unit 1704 can be operatively coupled toa memory 1706. The memory 1706 can include transitory and/ornon-transitory storage media configured to store data, and/or logic,code, and/or program instructions executable by the processing unit 1704for performing one or more routines or functions. The memory can includeone or more memory units (e.g., removable media or external storage suchas an SD card or random access memory (RAM)). In some embodiments, datafrom the input module 1702 can be directly conveyed to and stored withinthe memory units of the memory 1706. The memory units of the memory 1706can store logic, code and/or program instructions executable by theprocessing unit 1704 to perform any suitable embodiment of the methodsdescribed herein. For example, the processing unit 1704 can beconfigured to execute instructions causing one or more processors of theprocessing unit 1704 to process and display sensing data (e.g., images)received from the movable object, generate control commands includingnavigation commands and target information based on user input, causethe communication module 1710 to transmit and/or receive data, and thelike. The memory units can store sensing data or other data receivedfrom external devices (e.g., movable object). In some embodiments, thememory units of the memory 1706 can be used to store the processingresults produced by the processing unit 1704. Although FIG. 17 depicts asingle processing unit 1704 and a single memory 1706, one of skill inthe art would appreciate that this is not intended to be limiting, andthat the system 1700 can include a plurality of processing units and/ormemory units of the memory.

In some embodiments, the display module 1708 can be configured todisplay information of the movable object, carrier, and/or payload withrespect to position, translational velocity, translational acceleration,orientation, angular velocity, angular acceleration, or any suitablecombinations thereof. The display module 1708 can be configured todisplay information received from the movable object and/or payload suchas sensing data (e.g., images recorded by a camera or other imagecapturing device), tracking data such as described herein, controlfeedback data, and the like. In some embodiments, the display module1708 may be implemented by the same device that implements the inputmodule 1702. In other embodiments, the display module 1708 may beimplemented by a device that is separate from (but that may beoperatively coupled to) the device that implements the input module1702.

The communication module 1710 can be configured to transmit and/orreceive data from one or more remote devices (e.g., movable object,payload, base station, and the like). For example, the communicationmodule 1710 can be configured to transmit control data (e.g., navigationcommands, target information, and tracking control commands) to externalsystems or devices such as movable objects, carriers, and/or payloads.The communication module 1710 can also be configured to receive data(e.g., sensing data and tracking information) from such external systemsor devices. In some embodiments, the communication module 1710 caninclude a transmitter 1712 and a receiver 1714 respectively configuredto transmit and receive data to and from remote devices. In someembodiments, the communication module can include a transceiver thatcombines the functionalities of the transmitter and the receiver. Insome embodiments, the transmitter and the receiver can communicate witheach other as well as with the processing unit 1704. Any suitable meansof communication can be used, such as wired communication or wirelesscommunication described herein.

The tracking methods and systems as described herein can have a varietyof applications. For example, FIG. 18 illustrates an exemplary use casefor the present invention, where a user 1802 can configure a UAV 1804 toautomatically track herself while she engages in a variety ofactivities. As illustrated in FIG. 18, the user can provide targetinformation the UAV 1804 using a control terminal 1806. The controlterminal 1806 can provide a user interface for receiving user input forgenerating input information, such as described herein. For example, theuser interface may be configured to display one or more images (e.g.,pictures or videos) as captured by an imaging device 1805 onboard theUAV or elsewhere. The imaging device 1805 may be coupled to the UAV viaa carrier 1803 such as described herein. The images may have beentransmitted in real time or substantial real time from the UAV to thecontrol terminal. Alternatively, the images may be previously stored inthe control terminal and/or provided or generated by other sources. Someof the images may show the user. The user may select herself from one ofthese images, for example, by touching or otherwise selecting an area ofthe display that corresponds to the user. In some cases, the user mayspecify target type information instead of selecting specific targetinformation. Such target type information may be used to describe thetype of targets that should be tracked such as “a person with red hat.”The target information as specified by the user may be provided to theUAV. The transmission may occur via a wired or wireless connection. Invarious embodiments, the target information may be provided before orduring the operation of the UAV. For example, the target information maybe provided when the UAV is airborne or landed. In some embodiments, theuser may also specify the start time, end time, and/or duration of thetracking. Such timing information may be used to control the timing ofthe tracking process.

The target information may be used by processor(s) onboard the UAV 1804to automatically track the user 1802, for example, when the user isexercising (e.g., biking or running). The user may move about to staysubstantially still while being tracked, while the image device 1805onboard the UAV captures images of the user such that the resultingimages maintains a predetermined size or position of the user. Forexample, the position, attitude, orientation, velocity, zoom, and/orother characteristics of the UAV 1804, carrier 1803, and/or imagingdevice 1805 may be adjusted in real time to keep track of the user. Suchadjustment may be generated automatically by the processor(s) onboardthe UAV and without user intervention. Alternatively or additionally,the tracking can be performed with some user intervention (e.g.,adjustment of the zoom of the imaging device). The predetermined size orposition of the user may be specified by the user or according to systemdefault. In some embodiments, the user may wear or otherwise carry anobject to facilitate the tracking of the user. Such an object mayinclude, for example, bright colored cap or clothes.

In some embodiments, the images captured during tracking may betransmitted from the imaging device or UAV back to the control terminalor other suitable devices for display, playback, storage, edit, or otherpurposes. Such transmission may occur in real time or substantially realtime as the images are captured by the imaging device. Alternatively,there may be a substantial delay between the capture and thetransmission of the images. In some embodiments, the images are storedin a memory onboard the UAV without being transmitted anywhere. The usermay view such images in real time and adjust the target information orother aspects of the UAV or a component thereof, if necessary. Theadjusted target information may be provided to UAV as discussed aboveand the iterative process may continue until the desired images areobtained. In some embodiments, the images may be stored for laterviewing or playback. In some embodiments, the images may be transmittedfrom UAV, imaging device and/or control terminal to some remote server.For example, the images may be shared on a social network platform suchas Facebook or Twitter.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A method of tracking a target with an imagingdevice onboard an unmanned aerial vehicle (UAV), the method comprising:identifying one or more targets in a first image captured by the imagingdevice, the imaging device coupled to the UAV via a carrier configuredto permit the imaging device to move relative to the UAV; receiving auser selection of a target to be tracked from the one or more targets inthe first image, wherein the first image is displayed on a remotecontroller; generating target information for the target to be trackedbased on the user selection of the target, the target informationcomprising: initial information generated based on coordinates and pixelsize of the target in the first image, the initial information includingan initial target position of the target in the first image and aninitial target size of the target in the first image; and expectedinformation generated based on information about the target andspecifying characteristics that the target is expected to meet in imagescaptured by the imaging device, the characteristics including a targetposition of the target in the images captured by the imaging deviceand/or a target size of the target in the images captured by the imagingdevice, and the expected information including: an expected targetposition of the target in the images captured by the imaging device, theexpected target position being away from the initial target position bya predetermined offset; and/or an expected target size of the target inthe images captured by the imaging device, the expected target sizeequaling the initial target size multiplied by a predetermined ratio;detecting the target in a second image captured by the imaging devicebased on the initial information; detecting a deviation of a detectedposition and/or a detected size of the target in the second image fromthe expected target position and/or the expected target size; andadjusting at least one of the UAV or the carrier based on the deviation,to reduce the deviation to maintain the target to have a relativeposition and/or a relative size within a field of view of the imagedevice that corresponds to the expected target position and/or theexpected target size.
 2. The method of claim 1, wherein the deviation isrelated to a change in position of the target and a calculatedadjustment is related to an angular velocity of the imaging device atleast relative to a pitch axis of the imaging device.
 3. The method ofclaim 2, wherein the calculated adjustment is configured to generatecontrol signals for the carrier so as to cause the imaging device tomove relative to the UAV.
 4. The method of claim 1, further comprisestransmitting, in real-time, the images captured by the imaging device tothe remote controller.
 5. The method of claim 1, wherein the targetinformation further comprises target type information comprising color,texture, or pattern information.
 6. The method of claim 5, whereinidentifying the one or more targets in the first image captured by theimaging device comprises identifying, based on the target typeinformation, using an image recognition algorithm.
 7. The method ofclaim 1, wherein adjusting the at least one of the UAV or the carrierbased on the deviation includes adjusting the at least one of the UAV orthe carrier at an adjustment speed proportional to the deviation.
 8. Asystem for controlling an unmanned aerial vehicle (UAV), comprising: oneor more receivers configured to receive from a remote controller one ormore navigation commands to move the UAV along a flight path; an imagingdevice coupled to the UAV via a carrier configured to permit the imagedevice to move relative to the UAV, and one or more processorsconfigured to: receive a user selection of a target in a first imagedisplayed on the remote controller; generate target information for thetarget to be tracked based on the user selection of the target, thetarget information comprising: initial information generated based oncoordinates and pixel size of the target in the first image, the initialinformation including an initial target position of the target in thefirst image and an initial target size of the target in the first image;and expected information generated based on information about the targetand specifying characteristics that the target is expected to meet inimages captured by the imaging device, the characteristics including atarget position of the target in the images captured by the imagingdevice and/or a target size of the target in the images captured by theimaging device, and the expected information including: an expectedtarget position of the target in the images captured by the imagingdevice, the expected target position being away from the initial targetposition by a predetermined offset; and/or an expected target size ofthe target in the images captured by the imaging device, the expectedtarget size equaling the initial target size multiplied by apredetermined ratio; detect the target in a second image captured by theimaging device based on the initial information; detect a deviation of adetected position and/or a detected size of the target in the secondimage from the expected target position and/or the expected target size;and adjust at least one of the UAV or the carrier based on the deviationto reduce the deviation to maintain the target to have a relativeposition and/or a relative size within a field of view of the imagedevice that corresponds to the expected target position and/or theexpected target size.
 9. The system of claim 8, wherein the deviation isrelated to a change in position of the target and a calculatedadjustment is related to an angular velocity of the imaging device atleast relative to a pitch axis of the imaging device.
 10. The system ofclaim 8, wherein the one or more processors are further configured totransmit, in real-time, images captured by the imaging device to theremote controller.
 11. The system of claim 8, wherein the user selectionof the target is achieved by a user selecting an area of the first imagebeing displayed on a display, the selected area corresponding to thetarget.
 12. The system of claim 11, wherein the one or more processorsare further configured to display, on the display, the selected targetwith a selection indicator in response to the user selection of thetarget, the selection indicator indicating that the target has beenselected by the user.
 13. An unmanned aerial vehicle (UAV) with visualtracking capability, the UAV comprising: a carrier configured to carryan imaging device; one or more receivers configured to receive from aremote controller one or more navigation commands to move the UAV alonga flight path, and one or more processors configured to: generate targetinformation for the target to be tracked based on a user selection ofthe target, the target information comprising: initial informationgenerated based on coordinates and pixel size of the target in the firstimage, the initial information including an initial target position ofthe target in the first image and an initial target size of the targetin the first image; and expected information generated based oninformation about the target and specifying characteristics that thetarget is expected to meet in images captured by the imaging device, thecharacteristics including a target position of the target in the imagescaptured by the imaging device and/or a target size of the target in theimages captured by the imaging device, and the expected informationincluding: an expected target position of the target in the imagescaptured by the imaging device, the expected target position being awayfrom the initial target position by a predetermined offset; and/or anexpected target size of the target in the images captured by the imagingdevice, the expected target size equaling the initial target sizemultiplied by a predetermined ratio; detect the target in a second imagecaptured by the imaging device based on the initial information; detecta deviation of a detected position and/or a detected size of the targetin the second image from the expected target position and/or theexpected target size; and adjust at least one of the UAV or the carrierbased on the deviation to reduce the deviation to maintain the target tohave a relative position and/or a relative size within a field of viewof the image device that corresponds to the selected position and/or theexpected target size.
 14. The UAV of claim 13, wherein the deviation isrelated to a change in position of the target and a calculatedadjustment is related to an angular velocity of the imaging device atleast relative to a pitch axis of the imaging device.
 15. The UAV ofclaim 13, wherein the one or more processors are further configured totransmit, in real-time, images captured by the imaging device to theremote controller.
 16. The UAV of claim 13, wherein the user selectionof the target is achieved by a user selecting an area of the first imagebeing displayed on a display, the selected area corresponding to thetarget.